System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to capture an image of the contents of a refrigerator; image recognition means for identifying the captured image to identify a food item; means for suggesting a recipe based on information of the identified food item; and means for providing detailed information of the suggested recipe.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] One of the challenges facing working women, especially working mothers, is the time and effort required to prepare dinner after work, which can be a major source of stress. In particular, given the current situation in which women often cook dinner, even though household chores are shared, and children's meals must also be considered, there is a need for a means to quickly and effectively decide on a dinner menu. However, there is currently no system that can quickly suggest optimal recipes based on the contents of the refrigerator. The present invention aims to solve this problem and reduce the stress of preparing dinner for users. [Means for solving the problem]

[0005] The present invention provides a system including a means for a user to acquire an image of the contents of a refrigerator, an image recognition means for identifying food items by recognizing the acquired image, a means for suggesting recipes based on information about the identified food items, and a means for providing detailed information about the suggested recipes. The system also includes a means for prioritizing recipes by taking into account the user's past selections and preferences, and a means for receiving feedback from the user and improving the accuracy of the recipe suggestions. This allows the user to easily select the optimal dinner menu based on the contents of the refrigerator, significantly reducing the effort and stress of making dinner.

[0006] "User" refers to an individual who uses this system to take a photo of the contents of their refrigerator and receive recipe suggestions.

[0007] "Means for acquiring an image of the contents of the refrigerator" refers to a function that allows a user to take a picture of the contents of the refrigerator using a smartphone or other photographic device and acquire the image data.

[0008] "Image recognition means" refers to the algorithms and software used to analyze the captured image and identify the food items in the image.

[0009] "Food items" refer to specific ingredients or foods stored in the refrigerator, and are identified by image recognition means.

[0010] The "means for suggesting recipes" refers to a function that presents appropriate cooking recipes to the user based on information about the identified food items.

[0011] "Means for providing detailed information" refers to a function for providing users with detailed information such as cooking instructions, calorie information, and nutritional balance of the suggested recipe.

[0012] "Prioritization means" refers to algorithms or software that prioritize suggested recipes based on a user's past choices and preferences.

[0013] "Means for receiving feedback and improving accuracy" refers to functionality for collecting feedback information from users and analyzing that information to improve the accuracy of future recipe suggestions. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] As an embodiment of this invention, we provide a system that allows a user to scan the contents of their refrigerator using a smartphone or tablet, identifies food items using image recognition technology, and suggests optimal recipes based on that information. Next, we will explain the program's processing in natural language using concrete examples.

[0036] User scans the contents of the refrigerator

[0037] The user launches the "Dinner Omakase" app installed on their smartphone. Using the app's camera function, the user takes a picture of the contents of the refrigerator. The smartphone acquires the image and sends it to the server.

[0038] The server performs image recognition

[0039] The server receives the image of the refrigerator interior sent by the user. A deep learning-based image recognition module is installed on the server, and this module is used to recognize food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. The list of identified food items is stored in a database.

[0040] Server suggests recipes

[0041] The server searches a recipe database based on the identified food item list. The recipe database contains a vast amount of recipe information, and searches for and suggests recipes related to the specific food items. The server then filters and prioritizes the search results, taking into account the user's past selection history and preferences. For example, recipes such as "grilled chicken and broccoli" and "pasta with tomato and chicken" are suggested.

[0042] The device notifies the user of recipe suggestions

[0043] The server generates a list of suggested recipes and sends it to the user's device. The device's app displays the list of suggested recipes on the user's screen and sends the following interactive message:

[0044] We recommend the following recipes for dinner tonight:

[0045] 1. Grilled Chicken and Broccoli

[0046] 2. Tomato and chicken pasta

[0047] Which one would you like?

[0048] User selects a recipe and receives details

[0049] The user selects one of the suggested recipes. For example, if the user selects "grilled chicken and broccoli," the device sends the selection to the server. The server generates detailed information about the selected recipe and sends it to the device. The detailed information includes cooking instructions, a list of ingredients, calorie information, and nutritional balance. The device displays this information on the user's screen.

[0050] Users provide feedback

[0051] After trying the suggested recipes, users can provide feedback from within the app, such as "It was delicious" or "I wish it was a little saltier." The device then sends this feedback information to the server.

[0052] The server collects feedback and improves the algorithm

[0053] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[0054] As described above, this system efficiently utilizes the contents of the refrigerator and significantly reduces the effort required for users to make dinner. As a specific example, the user can select a recipe based on the items in the refrigerator and then recreate it at home, saving time and effort.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] The user launches the "Dinner Omakase" app on their smartphone and uses the app's camera function to take a picture of the contents of the refrigerator. The device temporarily saves the image as a "refrigerator image."

[0058] Step 2:

[0059] The terminal sends the "refrigerator interior image" to the server, and the image data is transferred to the server via the network.

[0060] Step 3:

[0061] The server receives the "refrigerator image" sent by the user. The received image data is passed to the image recognition module in the server.

[0062] Step 4:

[0063] The server's image recognition module uses deep learning-based algorithms to analyze and identify food items in the image, such as "tomato," "chicken," and "broccoli."

[0064] Step 5:

[0065] The server generates a list of the identified food items and stores it in a database. The identified food item information is stored in the database.

[0066] Step 6:

[0067] The server searches the recipe database based on the food item list to retrieve related recipes. Multiple recipes are found in the recipe database.

[0068] Step 7:

[0069] The server prioritizes the retrieved recipes based on the user's past selection history and preferences. For example, "grilled chicken and broccoli" and "tomato and chicken pasta" are listed as high-priority recipes.

[0070] Step 8:

[0071] The server generates a prioritized recipe list and transmits it to the user's terminal, where it is displayed in a format that the user can view.

[0072] Step 9:

[0073] The terminal will display a message to the user similar to:

[0074] We recommend the following recipes for dinner tonight:

[0075] 1. Grilled Chicken and Broccoli

[0076] 2. Tomato and chicken pasta

[0077] Which one would you like?

[0078] The user selects one of the suggested recipes.

[0079] Step 10:

[0080] The terminal receives the recipe selected by the user and transmits the information to the server.

[0081] Step 11:

[0082] The server generates detailed information about the selected recipe and sends it to the device, including cooking instructions, a list of ingredients, calorie information, and nutritional balance.

[0083] Step 12:

[0084] The device displays the detailed information received from the server to the user, allowing the user to check the specific cooking steps and ingredient information for the selected recipe.

[0085] Step 13:

[0086] After the user tries the recipe, they submit feedback from within the app. The device collects the user's feedback information and sends it to the server.

[0087] Step 14:

[0088] The server stores the received feedback in a database, analyzes the stored feedback information, and updates the recommendation algorithm to improve the accuracy of future recipe suggestions.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] Conventional recipe suggestion systems have problems in that they do not allow users to fully utilize the ingredients in their refrigerators, and they are unable to suggest recipes that fully reflect the user's preferences and past selection history. They also lack a means to improve the accuracy of recipe suggestions using user feedback. As a result, users spend a lot of time finding suitable recipes, and are unable to efficiently use the ingredients in their refrigerators.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes a means for a user to acquire an image of the contents of the refrigerator, an image recognition means for identifying the acquired image and identifying food items, a means for suggesting recipes based on information about the identified food items, a means for providing detailed information about the suggested recipes, a means for the user to provide feedback, and a means for accumulating the provided feedback and improving the recipe suggestion algorithm. This allows users to efficiently use ingredients in the refrigerator and receive recipe suggestions based on their individual preferences and past selection history. The accuracy of the recipe suggestions can also be improved based on the feedback.

[0094] A "user" is a person who uses the system to manage ingredients in the refrigerator and receive recipe suggestions.

[0095] "Means for acquiring images of the contents of the refrigerator" is a function that allows a user to take a photo of the food in the refrigerator using a smartphone or tablet and send the image data to the system.

[0096] "Image recognition means" refers to technology that analyzes acquired image data and identifies ingredients in the image. A deep learning-based algorithm is used.

[0097] "Food items" refers to ingredients or food stored in the refrigerator, such as tomatoes, chicken, and broccoli.

[0098] The "means for suggesting recipes" is a function that provides optimal cooking methods and cooking ideas based on information about the identified food items.

[0099] "Detailed information" refers to additional information about the suggested recipe, such as specific cooking instructions, a list of ingredients needed, calorie information, and nutritional balance.

[0100] "Feedback means" refers to a function that allows users to send ratings and comments on suggested recipes to the system.

[0101] "Means for accumulating feedback and improving the recipe suggestion algorithm" refers to a method for accumulating feedback data provided by users in a database and analyzing that data to improve the accuracy of the recipe suggestion algorithm.

[0102] "User's past choices and preferences" refers to the recipes and ingredients the user has chosen in the past, the types of dishes they like, and other information that is taken into consideration when suggesting recipes.

[0103] As an embodiment of the present invention, we provide a system in which a user scans the contents of their refrigerator using a smartphone or tablet, identifies food items using image recognition technology, and suggests optimal recipes based on that information. This system has the user, a terminal, and a server as its main elements, and these elements function in cooperation with each other.

[0104] User scans the contents of the refrigerator

[0105] The user launches the app installed on their smartphone. Using the app's camera function, the user takes a picture of the contents of the refrigerator. For example, imagine a scenario in which the user takes out "tomatoes," "chicken," and "broccoli" from the refrigerator and takes a picture of them. The smartphone acquires the captured image and sends it to the server via the HTTPS protocol.

[0106] The server performs image recognition

[0107] The server receives the image of the refrigerator interior sent by the user. The server is equipped with a deep learning-based image recognition module (e.g., TensorFlow, PyTorch) that recognizes food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. Information about the recognized food items is stored in a database (e.g., MySQL, MongoDB).

[0108] Server suggests recipes

[0109] The server searches a recipe database based on the recognized food item list. The recipe database contains a vast amount of recipe information, and searches for and suggests recipes related to specific food items. The server also takes into account the user's past selection history and preferences to filter and prioritize the search results. For example, recipes such as "grilled chicken and broccoli" and "pasta with tomato and chicken" are suggested.

[0110] The device notifies the user of recipe suggestions

[0111] The server sends the generated recipe list to the user's device. The smartphone app displays the suggested recipe list on the user's screen and notifies the user with an interactive message:

[0112] We recommend the following recipes for dinner tonight:

[0113] 1. Grilled Chicken and Broccoli

[0114] 2. Tomato and chicken pasta

[0115] Which one would you like?

[0116] User selects a recipe and receives details

[0117] The user selects one of the suggested recipes. For example, if the user selects "grilled chicken and broccoli," the smartphone sends the selection to the server. The server generates detailed information about the selected recipe and sends it to the device. The detailed information includes cooking instructions, a list of ingredients, calorie information, and nutritional balance. The device displays this information on the user's screen.

[0118] Users provide feedback

[0119] After trying the suggested recipes, users can provide feedback from within the app, such as "It was delicious" or "I wish it was a little saltier." The device then sends this feedback information to the server.

[0120] The server collects feedback and improves the algorithm

[0121] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[0122] Example prompts for generative AI models

[0123] Prompt statement:

[0124] "Design a system that analyzes a user's image of the inside of a refrigerator, identifies the food items contained within, and suggests optimal recipes based on that. Include recipe filtering that takes into account the user's past choices and preferences."

[0125] In this way, the system efficiently utilizes the contents of the refrigerator and significantly reduces the effort required for users to make dinner. For example, users can select a recipe based on the items in the refrigerator and then recreate it at home, saving time and effort.

[0126] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0127] Step 1:

[0128] A user scans the contents of the refrigerator with their smartphone

[0129] Input: App installed on smartphone, user's shooting operation

[0130] How it works: The user launches the app and uses the camera to take a picture of the food in the refrigerator. They can also take multiple images in succession.

[0131] Output: Image data of the inside of the refrigerator

[0132] Step 2:

[0133] The device sends the image to the server

[0134] Input: Captured image, smartphone network connection

[0135] What it does: The app sends the acquired image data to a server using the HTTPS protocol.

[0136] Output: Image data sent to the server

[0137] Step 3:

[0138] The server receives and stores the image

[0139] Input: Submitted image data

[0140] What it does: The server saves the image data in a temporary directory, specifically as a file, for later processing.

[0141] Output: Saved image file

[0142] Step 4:

[0143] The server performs image recognition

[0144] Input: Saved image file

[0145] How it works: The server's deep learning-based image recognition module analyzes the image and identifies the food item, using a Convolutional Neural Network (CNN) model (e.g., TensorFlow or PyTorch).

[0146] Output: A list of identified food items (e.g., "tomato," "chicken," "broccoli")

[0147] Step 5:

[0148] The server stores the food item list in a database.

[0149] Input: A list of identified food items

[0150] What it does: Stores the identified food item information in JSON format in a database (e.g. MySQL, MongoDB).

[0151] Output: Stored food item data

[0152] Step 6:

[0153] The server searches for and suggests recipes

[0154] Input: Stored food item data

[0155] How it works: Searches for recipes related to a specified food item in a database, filters the search results based on past history and preferences, and extracts relevant recipes using SQL queries.

[0156] Output: A list of suggested recipes (e.g. "Grilled Chicken and Broccoli" or "Pasta with Tomato and Chicken")

[0157] Step 7:

[0158] The device notifies the user of the recipe list

[0159] Input: Suggested recipe list

[0160] Operation: The smartphone app displays the recipe list received from the server in the form of a message to the user.

[0161] Output: Recipe selection message screen

[0162] Step 8:

[0163] The user selects a recipe

[0164] Input: User selection of recipe

[0165] How it works: The user selects one of the suggested recipes on the screen, and the selection is sent to the server.

[0166] Output: The ID of the selected recipe.

[0167] Step 9:

[0168] The server generates and sends the recipe details.

[0169] Input: ID of the selected recipe

[0170] What it does: Retrieves detailed information about the selected recipe from the recipe database (e.g., cooking instructions, ingredient list, calorie information, etc.) and sends the information in JSON format to the smartphone.

[0171] Output: Recipe details

[0172] Step 10:

[0173] The device displays detailed information to the user

[0174] Input: Recipe details received from the server

[0175] What it does: The app parses the details and displays them on the screen in a user-friendly format.

[0176] Output: Recipe details screen displayed

[0177] Step 11:

[0178] Users provide feedback

[0179] Input: User-initiated feedback input

[0180] How it works: The user enters feedback about a recipe and the app sends that data to the server.

[0181] Output: Feedback data sent

[0182] Step 12:

[0183] The server collects feedback and improves the algorithm

[0184] Input: Received feedback data

[0185] How it works: The server stores the feedback data in a database, analyzes it, and improves the recipe suggestion algorithm. It also retrains the machine learning model to reflect this in future recipe suggestions.

[0186] Output: An improved recipe suggestion algorithm

[0187] (Application example 1)

[0188] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0189] In recent years, food waste and shortening cooking times at home have become social issues. There is a demand for efficient use of refrigerator contents and reduction of waste, but current technology requires the time-consuming manual management of food in the refrigerator, making it difficult to find appropriate recipes based on that information. Furthermore, if necessary ingredients are in short supply, users have to go shopping, which is a significant burden on them. There is a need for a system that can solve these issues.

[0190] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0191] In this invention, the server includes means for a user to acquire an image of the contents of the refrigerator, image recognition means for identifying the acquired image and specifying food items, means for suggesting recipes based on information about the identified food items, means for providing detailed information about the suggested recipes, and means for automatically listing missing ingredients and ordering the ingredients from a delivery service, thereby enabling efficient use of the contents of the refrigerator and saving time and effort.

[0192] A "user" is an individual or group that uses this system to manage the contents of their refrigerator and receive recipe suggestions.

[0193] A "refrigerator" is a household or commercial appliance used to store food at the appropriate temperature.

[0194] "Images" are photos of food in the refrigerator taken by the user using a smartphone or tablet.

[0195] "Acquisition" means that a user takes a picture of the inside of the refrigerator using a smartphone or tablet and imports that image data into the system.

[0196] "Ingredient items" are specific foods and ingredients stored in the refrigerator.

[0197] "Image recognition" is a technology that analyzes captured images and identifies the food items contained within them.

[0198] "Recipe suggestion" refers to providing a cooking method suitable for the user based on information about the identified ingredient items.

[0199] "Detailed information" refers to information about the proposed recipe, including specific cooking steps, necessary ingredients, calorie information, and the like.

[0200] A "missing ingredient" is an ingredient item that is needed to create a suggested recipe but is not present in the refrigerator.

[0201] "Delivery service" is a service that allows users to order missing materials online and have these materials delivered promptly.

[0202] "Automatic Listing" is a function that automatically generates a list of ingredients that are in short supply based on recipe suggestions.

[0203] An "order" is a series of actions and procedures in which a user requests a delivery service to provide missing materials and arranges for delivery.

[0204] To implement this invention, a user uses a smartphone to scan the contents of their refrigerator, and the server uses image recognition technology to identify food items and suggests optimal recipes based on that information. Furthermore, the system automatically lists any missing ingredients and allows the user to order those ingredients via a delivery service.

[0205] Hardware and Software Configuration

[0206] Hardware:

[0207] On the user's phone:

[0208] Use the camera feature to scan the contents of your refrigerator.

[0209] software:

[0210] server:

[0211] Use Deep Learning-based frameworks such as TensorFlow / Keras for image recognition.

[0212] PIL (Python Imaging Library) is used for image processing.

[0213] The requests library is used to process HTTP requests.

[0214] Terminal app:

[0215] The contents of the refrigerator are scanned and the images are sent to a server.

[0216] The suggested recipe and the list of ingredients that are in short supply received from the server are displayed to the user.

[0217] Call the delivery service API to order the required ingredients.

[0218] System Operation

[0219] Fridge scanning and image recognition

[0220] The user launches the "Refrigerator-Linked Delivery Assistant" app on their smartphone and uses the camera to take a picture of the food in their refrigerator. The image data is sent from the device app to the server. The server uses a deep learning-based image recognition system to identify the food items in the image and generate a list of identified items. For example, "tomatoes," "chicken," and "onions" in the refrigerator are recognized.

[0221] Recipe suggestions and a list of missing ingredients

[0222] Based on the recognized food item list, the server searches a recipe database and suggests several recipes suitable for the user, taking into account the user's past selection history and preferences. The server also automatically lists any ingredients needed for the suggested recipes that are missing from the refrigerator. For example, a recipe for "Tomato Chicken Salad" may be suggested, but the server will point out that the required "lettuce" is missing.

[0223] Automating delivery orders

[0224] The list of missing ingredients is sent from the device app to the delivery service API, which automatically places an order, allowing users to get the missing ingredients immediately. Once the delivery order is successfully processed, the order status is displayed on the device.

[0225] Specific examples

[0226] For example, suppose a user scans the "tomatoes," "chicken," and "onions" in their refrigerator. Based on this, the server suggests a recipe for "Tomato Chicken Salad" and lists the missing ingredient, "lettuce," which is needed but not present. The server then automatically orders "lettuce" using a delivery service API, and the user can receive the ingredients via delivery.

[0227] Prompt Sentence Examples

[0228] text

[0229] Scan the image of the refrigerator

[0230] recipes, missing_items = scan_fridge('fridge_image.jpg')

[0231] Create a delivery order

[0232] if missing_items:

[0233] order_status = order_missing_items(missing_items)

[0234] print("Order Status:", order_status)

[0235] View suggested recipes

[0236] if recipes:

[0237] for idx, recipe in enumerate(recipes):

[0238] print(f"Recipe {idx + 1}:{recipe['name']}")

[0239] print(f"ingredients:{recipe['ingredients']}")

[0240] print(f"Steps:{recipe['steps']}")

[0241] print("\n")

[0242] In this way, by efficiently managing food items in the refrigerator and easily obtaining the necessary ingredients, users can significantly save time and effort in cooking.

[0243] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0244] Step 1:

[0245] The user launches the "Refrigerator-Linked Delivery Assistant" app on their smartphone and uses the camera to take a picture of the inside of the refrigerator. The image is then temporarily saved in the smartphone's local storage.

[0246] Input: Image of the inside of the refrigerator taken with a smartphone camera

[0247] Output: Images saved in the smartphone's local storage

[0248] Specific operation: The user launches the app, takes a picture of the inside of the refrigerator with their smartphone camera, and saves the image.

[0249] Step 2:

[0250] The device app sends the saved image to the server, which receives it as an HTTP request.

[0251] Input: A picture of a refrigerator stored on a smartphone

[0252] Output: Image data sent to the server

[0253] Specific operation: The terminal application obtains the image data and sends the image to the server via an HTTP request.

[0254] Step 3:

[0255] The server inputs the received image data into a deep learning-based image recognition system to identify food items, and the list of identified items is stored in a database on the server.

[0256] Input: Image data sent to the server

[0257] Output: A list of identified food items

[0258] What it does: The server uses TensorFlow / Keras to analyze the image and identify the food items in it, resulting in a list of items.

[0259] Step 4:

[0260] The server searches a recipe database based on the recognized food item list and suggests multiple recipes suitable for the user. The suggested recipes are generated as a list, taking into account the user's past selection history and preferences.

[0261] Input: List of identified food items

[0262] Output: A list of suggested recipes

[0263] What it does: The server runs a database query to find relevant recipes and prioritizes them based on the user's historical data.

[0264] Step 5:

[0265] The server creates a list of missing ingredients based on the proposed recipe and sends it to the device app, which then passes the list to the delivery service API.

[0266] Input: Suggested recipe list

[0267] Output: List of missing materials

[0268] What it does: The server matches the suggested recipe with the current food items, lists any missing ingredients, and sends them to the delivery service API.

[0269] Step 6:

[0270] The terminal app displays the list of missing ingredients received from the server to the user, automatically places a delivery service order, and notifies the user of the order status of the missing ingredients.

[0271] Input: List of materials in short supply received from the server

[0272] Output: The list displayed to the user and the order status

[0273] What it does: The device app displays a list of missing ingredients and updates the user on the status of their delivery order in real time.

[0274] Step 7:

[0275] The user selects a suggested recipe and receives detailed information about it on their device, including cooking instructions, ingredients needed, and calorie information.

[0276] Input: Suggested recipe list

[0277] Output: Detailed information about the selected recipe

[0278] Specific behavior: The user selects a suggested recipe within the app and receives detailed information from the server.

[0279] Step 8:

[0280] After cooking, users provide feedback through the device app, which is then sent to the server and used to suggest recipes for future meals.

[0281] Input: User feedback

[0282] Output: Feedback data stored on the server

[0283] How it works: The user enters feedback into the device app and sends it to the server, which stores it in a database and uses it to improve the accuracy of the algorithm.

[0284] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0285] This invention provides a system that allows a user to scan the contents of their refrigerator using a smartphone or tablet, and then combines image recognition technology with emotion recognition technology to identify food items and suggest optimal recipes based on the user's emotional state. Below, the program's processing is explained in natural language using concrete examples.

[0286] User scans the contents of the refrigerator

[0287] The user launches the "Dinner Omakase" app installed on their smartphone. They then use the app's camera function to take a picture of the contents of their refrigerator. The device then acquires the captured image data (image of the inside of the refrigerator) and sends this image to the server.

[0288] The server performs image recognition

[0289] The server receives the image of the refrigerator interior sent by the user. A deep learning-based image recognition module is installed on the server, and this module is used to recognize food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. The list of identified food items is stored in a database.

[0290] The server performs emotion recognition.

[0291] When a user is using a device with a camera, the device captures the user's facial expression data and sends the data to an emotion recognition engine in the server. The emotion recognition engine analyzes the facial expression data and identifies the user's emotional state (e.g., "satisfied," "fatigued," "stressed," etc.).

[0292] Server suggests recipes

[0293] The server searches a recipe database based on the identified food item list and emotion recognition results to retrieve relevant recipes. The recipe database contains a vast amount of recipe information, and searches for and suggests recipes that are suitable for the identified food items and the user's emotional state. For example, if the user is in a "fatigue" state, recipes that are easy to make and can be made in a short time are prioritized.

[0294] The device notifies the user of recipe suggestions

[0295] The server generates a list of suggested recipes and sends it to the user's device. The device's app displays the list of suggested recipes on the user's screen and sends the following interactive message:

[0296] We recommend the following recipes for dinner tonight:

[0297] 1. Grilled Chicken and Broccoli (Cooking time: 20 minutes)

[0298] 2. Tomato and Chicken Pasta (Cooking time: 15 minutes)

[0299] Which one would you like?

[0300] The user selects one of the suggested recipes.

[0301] User selects a recipe and receives details

[0302] The device receives the recipe selected by the user and sends that information to the server. The server generates detailed information about the selected recipe and sends it to the device. The detailed information includes cooking instructions, a list of ingredients, calorie information, and nutritional balance. The device displays this information on the user's screen.

[0303] Users provide feedback

[0304] After trying the suggested recipes, users can provide feedback from within the app, such as "It was delicious" or "I wish it was a little saltier." The device then sends this feedback information to the server.

[0305] The server collects feedback and improves the algorithm

[0306] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[0307] In this way, by combining emotion recognition technology, the system can quickly suggest optimal recipes based on the user's current emotional state, significantly reducing the effort and stress of cooking dinner. As a specific example, a user can select a recipe based on the items in the refrigerator and their own emotional state, and then actually recreate it at home, saving time and effort.

[0308] The processing flow will be explained below.

[0309] Step 1:

[0310] The user launches the "Dinner Omakase" app on their smartphone and uses the app's camera function to take a picture of the contents of the refrigerator. The device temporarily saves the image as a "refrigerator image."

[0311] Step 2:

[0312] The terminal sends the "refrigerator interior image" to the server, and the image data is transferred to the server via the network.

[0313] Step 3:

[0314] The server receives the "refrigerator image" sent by the user. The received image data is passed to the image recognition module in the server.

[0315] Step 4:

[0316] The server's image recognition module uses deep learning-based algorithms to analyze and identify food items in the image, such as "tomato," "chicken," and "broccoli."

[0317] Step 5:

[0318] The server generates a list of the identified food items and stores it in a database. The identified food item information is stored in the database.

[0319] Step 6:

[0320] If the user is using a device with a camera, the device captures the user's facial expression data and sends the data to an emotion recognition engine.

[0321] Step 7:

[0322] The server's emotion recognition engine analyzes the facial expression data and identifies the user's emotional state (e.g., "satisfied," "fatigued," "stressed," etc.) This emotional state information is stored in a database.

[0323] Step 8:

[0324] The server searches the recipe database based on the food item list and emotion recognition results to retrieve related recipes. Multiple recipes are found in the recipe database.

[0325] Step 9:

[0326] The server prioritizes the retrieved recipes based on the user's past selection history, preferences, and current emotional state. For example, if the user is "fatigued," recipes that are easy to prepare and can be made quickly will be prioritized.

[0327] Step 10:

[0328] The server generates a prioritized recipe list and transmits it to the user's terminal, where it is displayed in a format that the user can view.

[0329] Step 11:

[0330] The terminal will display a message to the user similar to:

[0331] We recommend the following recipes for dinner tonight:

[0332] 1. Grilled Chicken and Broccoli (Cooking time: 20 minutes)

[0333] 2. Tomato and Chicken Pasta (Cooking time: 15 minutes)

[0334] Which one would you like?

[0335] The user selects one of the suggested recipes.

[0336] Step 12:

[0337] The terminal receives the recipe selected by the user and transmits the information to the server.

[0338] Step 13:

[0339] The server generates detailed information about the selected recipe and sends it to the device, including cooking instructions, a list of ingredients, calorie information, and nutritional balance.

[0340] Step 14:

[0341] The device displays the detailed information received from the server to the user, allowing the user to check the specific cooking steps and ingredient information for the selected recipe.

[0342] Step 15:

[0343] After the user tries the recipe, they submit feedback from within the app. The device collects the user's feedback information and sends it to the server.

[0344] Step 16:

[0345] The server stores the received feedback in a database, analyzes the stored feedback information, and updates the recommendation algorithm to improve the accuracy of future recipe suggestions.

[0346] Example 2

[0347] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0348] Conventional food management and recipe recommendation systems have difficulty not only accurately identifying the food in a user's refrigerator, but also automating optimal recipe recommendations that take into account the user's emotional state. This has left users unable to quickly choose an appropriate dish that suits their current mood while utilizing the food in their refrigerator, often resulting in stress when preparing dinner. To solve this problem, a recipe recommendation system that simultaneously considers the food in the refrigerator and the user's emotional state is needed.

[0349] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0350] In this invention, the server includes means for allowing a user to acquire an image of the contents of the refrigerator, image recognition means for identifying the acquired image and specifying food items, emotion recognition means for acquiring data on the user's facial expression and analyzing the data to identify the user's emotional state, means for suggesting recipes based on information on the identified food items and the user's emotional state, and means for providing detailed information on the suggested recipes. This makes it possible to quickly suggest optimal recipes based on information on the food items in the refrigerator and the user's current emotional state.

[0351] "User" refers to an individual who uses the system to scan the contents of their refrigerator and receive recipe suggestions.

[0352] "Means of acquiring images" refers to the process in which a user takes a photo of the food in the refrigerator using the camera function of a smartphone or other device.

[0353] "Image recognition means" refers to technology for analyzing acquired image data and identifying food items in the refrigerator.

[0354] "Facial expression data" refers to image data captured by a camera of a user's face, and is used for emotion recognition.

[0355] "Emotion recognition means" refers to technology that analyzes facial expression data and identifies the user's emotional state (e.g., "satisfaction," "fatigue," "stress," etc.).

[0356] "Means for suggesting recipes" refers to a function that selects and suggests appropriate recipes based on information about the identified food items and the user's emotional state.

[0357] "Means for providing detailed information" refers to a function that provides users with detailed information such as cooking instructions for the suggested recipe, a list of required ingredients, calorie information, and nutritional balance.

[0358] The present invention is a system that allows a user to scan the contents of a refrigerator using a smartphone or tablet, and combines image recognition technology and emotion recognition technology to identify food items and suggest optimal recipes based on the user's emotional state. Next, a specific embodiment of this system will be described.

[0359] The system uses the following hardware and software:

[0360] Mobile devices such as smartphones and tablets

[0361] Camera on the device

[0362] Deep learning-based image recognition modules (e.g., TensorFlow and PyTorch)

[0363] Emotion recognition engine (e.g., Microsoft Azure Face API)

[0364] Recipe Database

[0365] Specific steps

[0366] User scans the contents of the refrigerator

[0367] The user launches the "Dinner Omakase" app installed on their smartphone and uses the camera to take a picture of the contents of the refrigerator. The device acquires the captured image data and sends it to the server in real time.

[0368] The server performs image recognition

[0369] The server receives the image of the refrigerator sent by the user and uses a deep learning-based image recognition module to recognize food items in the image, such as "tomato," "chicken," and "broccoli," and stores this information in a database.

[0370] The server performs emotion recognition.

[0371] When a user is using a device with a camera, the device captures the user's facial expression data and sends it to an emotion recognition engine in the server. The server analyzes the facial expression data and identifies the user's emotional state (e.g., "satisfied," "fatigued," "stressed," etc.).

[0372] Server suggests recipes

[0373] The server searches the recipe database based on the identified food item list and the emotion recognition results to retrieve relevant recipes. For example, if the user is in a "fatigue" state, it will prioritize recipes that are easy to make and can be made in a short time.

[0374] The device notifies the user of recipe suggestions

[0375] The server generates a list of suggested recipes and sends it to the user's device, where the app displays the list on the screen and provides specific lists and options.

[0376] User selects a recipe and receives more information

[0377] The user selects one of the suggested recipes. The selected recipe information is sent from the device to the server, and the server generates detailed cooking instructions, a list of ingredients, calorie information, nutritional balance, and other information and sends it to the device. The device then displays this information on the user's screen.

[0378] Users provide feedback

[0379] After trying the suggested recipes, users provide feedback from within the app, which the device then captures and sends to the server.

[0380] The server accumulates feedback and improves the algorithm

[0381] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[0382] Examples of prompt statements

[0383] For example, if a user uses a "Dinner Picker" app to scan the items "clams" and "garlic" in the refrigerator, and the app recognizes the user's facial expression as indicating a "fatigue" state, the prompt sentence based on this example might be:

[0384] Use the "Dinner Omakase" app to scan the contents of your refrigerator (clams, garlic) and suggest the best dinner recipe through image and emotion recognition. The user's emotional state is "fatigue."

[0385] In this way, by combining emotion recognition technology, the system can quickly suggest optimal recipes based on the user's current emotional state, significantly reducing the effort and stress of cooking dinner.

[0386] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0387] Processing flow

[0388] Step 1:

[0389] The user launches the app installed on their smartphone and takes a picture of the contents of the refrigerator. At this time, the user uses the camera function in the app to capture image data (images of the food in the refrigerator). The image data obtained as input is saved in the device's internal storage. Specifically, the user taps the camera icon on the app screen and takes a picture of the inside of the refrigerator with the smartphone's camera.

[0390] Step 2:

[0391] The device sends the acquired image data to the server. At this time, the device's app uploads the image data to the server via an Internet connection. The input is the image data acquired in step 1, and the output is the image data sent to the server. Specifically, the device uploads the data to the server using a background process.

[0392] Step 3:

[0393] The server receives image data sent by the user and identifies food items using an image recognition module. The image data received as input is analyzed using a deep learning-based image recognition algorithm (e.g., TensorFlow or PyTorch) to generate a list of food items. Specifically, the server calls the image recognition API and generates a list of items such as "tomato," "chicken," and "broccoli."

[0394] Step 4:

[0395] The server stores the identified food item list in a database, where the input is the food item list obtained in step 3 and the output is the information stored in the database. Specifically, the server opens a database connection and executes an SQL query to store the listed food items.

[0396] Step 5:

[0397] When a user is using a device with a camera, the device acquires the user's facial expression data and sends it to the emotion recognition engine. The input is the user's facial expression image data, and the output is the transmission of facial expression data from the device to the server. Specifically, the user shows an expression in front of the camera, and the device captures it and uploads it to the server.

[0398] Step 6:

[0399] The server analyzes the received facial expression data using an emotion recognition engine to identify the user's emotional state. The input is facial expression data, and the output is the identified emotional state (e.g., "satisfied," "fatigued," "stressed," etc.). Specifically, the server calls an emotion recognition API (e.g., Microsoft Azure Face API) to analyze the emotional state.

[0400] Step 7:

[0401] The server searches the recipe database based on the identified food item list and emotion recognition results to suggest optimal recipes. The input is the food item list and the emotional state, and the output is a list of related recipes. Specifically, the server generates a query and executes an SQL query to retrieve suitable recipes from the recipe database.

[0402] Step 8:

[0403] The server generates a list of suggested recipes and sends it to the user's device. The input is the recipe list, and the output is sending the recipe list to the device. Specifically, the server sends the generated recipe list to the device via push notification.

[0404] Step 9:

[0405] The device displays the suggested recipe list on the user's screen. The input is the recipe list sent from the server, and the output is the list displayed on the user's screen. In concrete terms, the app receives the recipe list and displays it using a UI component.

[0406] Step 10:

[0407] The user selects one of the suggested recipes. The input is the displayed recipe list, and the output is the selected recipe information. Specifically, the user selects a recipe by tapping on the screen.

[0408] Step 11:

[0409] The terminal sends the selected recipe information to the server. The input is the recipe information selected by the user, and the output is the sending of a recipe list to the server. In concrete terms, the terminal uploads the selected content to the server.

[0410] Step 12:

[0411] The server generates detailed information for the selected recipe and sends it to the terminal. The input is the selected recipe information, and the output is the detailed recipe information. Specifically, the server generates detailed cooking instructions and ingredient lists and sends them to the terminal.

[0412] Step 13:

[0413] The device displays detailed recipe information on the user's screen. The input is the detailed recipe information sent from the server, and the output is the detailed information displayed on the user's screen. Specifically, the app receives the detailed information and displays it as a description or list.

[0414] Step 14:

[0415] The user enters feedback within the app. The input is the user's feedback comment, and the output is saving the feedback information to the device. Specifically, the user writes a comment in the feedback form and taps the submit button.

[0416] Step 15:

[0417] The terminal sends feedback information to the server. The input is the feedback information provided by the user, and the output is the transmission of the feedback information to the server. In concrete terms, the terminal uploads the transmitted feedback to the server.

[0418] Step 16:

[0419] The server accumulates the feedback sent by the user in a database and reflects it in future recipe suggestions. The input is the feedback data, and the output is an updated recipe suggestion algorithm. Specifically, the server stores the feedback data in storage and uses it to improve the machine learning algorithm.

[0420] (Application example 2)

[0421] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0422] Conventional recipe suggestion systems could suggest recipes based on the contents of a user's refrigerator, but they could not take the user's emotional state into account. This made it difficult for users to find products and recipes that matched their emotions at any given time. The present invention aims to solve these problems by providing a system that detects a user's emotional state and suggests optimal products and recipes based on that emotional state.

[0423] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0424] In this invention, the server includes means for a user to acquire an image of the contents of the refrigerator, image recognition means for identifying food items by recognizing the acquired image, means for suggesting recipes based on information on the identified food items, means for providing detailed information on the suggested recipes, emotion recognition means for scanning the user's facial expression to identify their emotional state, means for recommending products based on the user's emotional state, and means for providing detailed information on the recommended products, thereby enabling the user to receive recommendations that are optimal for their emotional state at the time.

[0425] "User" refers to an individual who uses the system.

[0426] "Means for acquiring an image of the contents of the refrigerator" refers to a function that allows a user to take a picture of the contents of the refrigerator using the camera on a smartphone or tablet and acquire the image data.

[0427] "Image recognition means" refers to technology for analyzing captured image data and identifying food items present in the image.

[0428] "Food items" refer to ingredients or foods present in an image, including specific items such as tomatoes or chicken.

[0429] The "means for suggesting recipes" refers to a function that provides the user with appropriate cooking methods and menus based on information about food items identified by the image recognition means.

[0430] "Means for providing detailed information" refers to a function that presents the user with specific cooking steps, a list of ingredients, calorie information, etc. for a recipe.

[0431] "Emotion recognition means for scanning facial expressions to identify emotional states" refers to technology for analyzing a user's facial expression data and identifying their emotional state at that time (e.g., joy, fatigue, etc.).

[0432] "Means for recommending products" refers to a function that suggests appropriate products to users based on their emotional state and past behavioral history.

[0433] "Means for providing detailed information about recommended products" refers to a function that displays detailed information about recommended products, such as descriptions, prices, and reviews, to users.

[0434] The present invention relates to a system that allows users to scan the contents of their refrigerator using a smartphone or tablet, combines image recognition technology with emotion recognition technology to identify food items, and suggests optimal recipes and products based on the user's emotional state.

[0435] System configuration

[0436] The system consists of the following main components:

[0437] 1. User device: A device equipped with a camera and display, such as a smartphone or tablet.

[0438] 2. Server: A cloud service that performs image recognition, emotion recognition, and recipe and product suggestions.

[0439] 3. Database: Stores food items, emotional states, recipes, product information, user behavior history and feedback.

[0440] Program execution procedure

[0441] 1. User scans the contents of the refrigerator

[0442] The user launches the dedicated application and takes a picture of the contents of the refrigerator using the camera function of their smartphone or tablet. The device acquires the image data and sends it to the server.

[0443] 2. Image recognition by the server

[0444] The server receives the image of the refrigerator sent by the user. The server is equipped with a deep learning-based image recognition module, which recognizes food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. The list of identified food items is stored in a database.

[0445] 3. Emotion recognition

[0446] When a user uses a device with a camera, the device captures the user's facial expression data and sends it to the server. The emotion recognition engine in the server analyzes the facial expression data and identifies the user's emotional state. For example, emotional states such as "joy," "fatigue," and "stress" are analyzed.

[0447] 4. Recipe and product suggestions

[0448] The server searches the recipe and product database based on the identified food item list and emotion recognition results to retrieve related recipes and products. Related recipes and products are suggested based on the user's emotional state. For example, if the user is "fatigued," recipes that are easy to make in a short time and products that help them relax are suggested first.

[0449] 5. Notifications on user devices

[0450] The server generates suggested recipes and product lists and sends them to the user's device. The device app displays the suggested recipes and product lists on the user's screen and sends interactive messages. For example, the following message might be displayed:

[0451] Here are some products we recommend for you right now:

[0452] 1. Grilled Chicken and Broccoli (Cooking time: 20 minutes)

[0453] 2. Tomato and Chicken Pasta (Cooking time: 15 minutes)

[0454] 3. Stress Relief Ball

[0455] 4. Herbal tea

[0456] Which one would you like?

[0457] Hardware and software used

[0458] Hardware:

[0459] Smartphone or tablet camera: Used to capture images of the user's facial expressions and the inside of the refrigerator.

[0460] software:

[0461] Python: The implementation language for the entire program.

[0462] opencv-python: Used to process facial expression and fridge image data.

[0463] Pillow: A library for image processing.

[0464] numpy: Used to process data.

[0465] TensorFlow: Emotion recognition using pre-trained models.

[0466] Prompt Sentence Examples

[0467] Write code to analyze a user's emotional state using facial images and recommend products that best match their emotions. Use the camera function to capture facial images and use TensorFlow to recognize emotions with a pre-trained model. Write a set of Python code to display a list of products based on the recognized emotion.

[0468] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0469] Step 1:

[0470] The user launches an application on their smartphone or tablet and uses the camera function to take a picture of the contents of the refrigerator. The device acquires the captured image data (image of the inside of the refrigerator) and sends this image to the server. The input is the image data captured by the user, and the output is the transfer of the image data to the server.

[0471] Step 2:

[0472] The server receives an image of the inside of the refrigerator sent by the user. It uses a deep learning-based image recognition module to recognize food items in the image. The input is image data of the inside of the refrigerator, and the output is a list of identified food items. Specifically, the image data is preprocessed and input into a model to identify food items.

[0473] Step 3:

[0474] The identified food item list is saved in a database. The input is the food item list output from image recognition, and the output is the process of saving it to the database. Specifically, the type and quantity of food items are written to the database.

[0475] Step 4:

[0476] The user takes a photo of their own facial expression using the camera function of their smartphone or tablet. The device acquires the facial expression data and sends it to the server. The input is the facial expression image taken by the user, and the output is the transfer of the facial expression data to the server.

[0477] Step 5:

[0478] The server receives the facial expression images and analyzes the facial expression data using an emotion recognition engine. The input is the facial expression image data, and the output is the user's emotional state. Specifically, the facial expression data is preprocessed and input into an emotion recognition model to identify the emotional state.

[0479] Step 6:

[0480] The server searches the recipe and product databases based on the identified food item list and emotional state to retrieve related recipes and products. The input is the food item list and emotional state, and the output is a list of recommended recipes and products. Specifically, it extracts information that matches the conditions from the recipe database and product database.

[0481] Step 7:

[0482] The server generates a list of recommended recipes and products and sends it to the user's device. The input is the recommendation list, and the output is data transmission to the user's device. Specifically, the server formats the selected information and delivers it to the user's device.

[0483] Step 8:

[0484] The user device displays the suggested recipes and product list on the screen and presents interactive messages to the user. For example, it presents multiple options along with the message "Here are the products we recommend for you right now:" The input is the recommendation list data from the server, and the output is the display to the user. Specifically, the information is presented visually using a GUI.

[0485] Step 9:

[0486] The user checks the detailed information of the suggested recipes and products and selects a specific recipe or product. The input is the user's selection, and the output is the transmission of the selected data to the server. Specifically, the system detects the user's action of pressing the selection button and sends the selected information to the server.

[0487] Step 10:

[0488] The server receives the user's selection information, generates detailed information about the selected recipe or product, and sends it to the user's device. The input is the user's selection information, and the output is a response with detailed information. Specifically, the server retrieves detailed information from the database, formats it, and provides it to the user.

[0489] Step 11:

[0490] The user terminal displays detailed information on the screen and provides cooking instructions and product details. The input is the detailed information data sent from the server, and the output is the display to the user. Specifically, the detailed information is presented visually using a GUI.

[0491] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0492] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0493] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0494] [Second embodiment]

[0495] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0496] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0497] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0498] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0499] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0500] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0501] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0502] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0503] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0504] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0505] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0506] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0507] As an embodiment of this invention, we provide a system that allows a user to scan the contents of their refrigerator using a smartphone or tablet, identifies food items using image recognition technology, and suggests optimal recipes based on that information. Next, we will explain the program's processing in natural language using concrete examples.

[0508] User scans the contents of the refrigerator

[0509] The user launches the "Dinner Omakase" app installed on their smartphone. Using the app's camera function, the user takes a picture of the contents of the refrigerator. The smartphone acquires the image and sends it to the server.

[0510] The server performs image recognition

[0511] The server receives the image of the refrigerator interior sent by the user. A deep learning-based image recognition module is installed on the server, and this module is used to recognize food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. The list of identified food items is stored in a database.

[0512] Server suggests recipes

[0513] The server searches a recipe database based on the identified food item list. The recipe database contains a vast amount of recipe information, and searches for and suggests recipes related to the specific food items. The server then filters and prioritizes the search results, taking into account the user's past selection history and preferences. For example, recipes such as "grilled chicken and broccoli" and "pasta with tomato and chicken" are suggested.

[0514] The device notifies the user of recipe suggestions

[0515] The server generates a list of suggested recipes and sends it to the user's device. The device's app displays the list of suggested recipes on the user's screen and sends the following interactive message:

[0516] We recommend the following recipes for dinner tonight:

[0517] 1. Grilled Chicken and Broccoli

[0518] 2. Tomato and chicken pasta

[0519] Which one would you like?

[0520] User selects a recipe and receives details

[0521] The user selects one of the suggested recipes. For example, if the user selects "grilled chicken and broccoli," the device sends the selection to the server. The server generates detailed information about the selected recipe and sends it to the device. The detailed information includes cooking instructions, a list of ingredients, calorie information, and nutritional balance. The device displays this information on the user's screen.

[0522] Users provide feedback

[0523] After trying the suggested recipes, users can provide feedback from within the app, such as "It was delicious" or "I wish it was a little saltier." The device then sends this feedback information to the server.

[0524] The server collects feedback and improves the algorithm

[0525] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[0526] As described above, this system efficiently utilizes the contents of the refrigerator and significantly reduces the effort required for users to make dinner. As a specific example, the user can select a recipe based on the items in the refrigerator and then recreate it at home, saving time and effort.

[0527] The processing flow will be explained below.

[0528] Step 1:

[0529] The user launches the "Dinner Omakase" app on their smartphone and uses the app's camera function to take a picture of the contents of the refrigerator. The device temporarily saves the image as a "refrigerator image."

[0530] Step 2:

[0531] The terminal sends the "refrigerator interior image" to the server, and the image data is transferred to the server via the network.

[0532] Step 3:

[0533] The server receives the "refrigerator image" sent by the user. The received image data is passed to the image recognition module in the server.

[0534] Step 4:

[0535] The server's image recognition module uses deep learning-based algorithms to analyze and identify food items in the image, such as "tomato," "chicken," and "broccoli."

[0536] Step 5:

[0537] The server generates a list of the identified food items and stores it in a database. The identified food item information is stored in the database.

[0538] Step 6:

[0539] The server searches the recipe database based on the food item list to retrieve related recipes. Multiple recipes are found in the recipe database.

[0540] Step 7:

[0541] The server prioritizes the retrieved recipes based on the user's past selection history and preferences. For example, "grilled chicken and broccoli" and "tomato and chicken pasta" are listed as high-priority recipes.

[0542] Step 8:

[0543] The server generates a prioritized recipe list and transmits it to the user's terminal, where it is displayed in a format that the user can view.

[0544] Step 9:

[0545] The terminal will display a message to the user similar to:

[0546] We recommend the following recipes for dinner tonight:

[0547] 1. Grilled Chicken and Broccoli

[0548] 2. Tomato and chicken pasta

[0549] Which one would you like?

[0550] The user selects one of the suggested recipes.

[0551] Step 10:

[0552] The terminal receives the recipe selected by the user and transmits the information to the server.

[0553] Step 11:

[0554] The server generates detailed information about the selected recipe and sends it to the device, including cooking instructions, a list of ingredients, calorie information, and nutritional balance.

[0555] Step 12:

[0556] The device displays the detailed information received from the server to the user, allowing the user to check the specific cooking steps and ingredient information for the selected recipe.

[0557] Step 13:

[0558] After the user tries the recipe, they submit feedback from within the app. The device collects the user's feedback information and sends it to the server.

[0559] Step 14:

[0560] The server stores the received feedback in a database, analyzes the stored feedback information, and updates the recommendation algorithm to improve the accuracy of future recipe suggestions.

[0561] Example 1

[0562] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0563] Conventional recipe suggestion systems have problems in that they do not allow users to fully utilize the ingredients in their refrigerators, and they are unable to suggest recipes that fully reflect the user's preferences and past selection history. They also lack a means to improve the accuracy of recipe suggestions using user feedback. As a result, users spend a lot of time finding suitable recipes, and are unable to efficiently use the ingredients in their refrigerators.

[0564] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0565] In this invention, the server includes a means for a user to acquire an image of the contents of the refrigerator, an image recognition means for identifying the acquired image and identifying food items, a means for suggesting recipes based on information about the identified food items, a means for providing detailed information about the suggested recipes, a means for the user to provide feedback, and a means for accumulating the provided feedback and improving the recipe suggestion algorithm. This allows users to efficiently use ingredients in the refrigerator and receive recipe suggestions based on their individual preferences and past selection history. The accuracy of the recipe suggestions can also be improved based on the feedback.

[0566] A "user" is a person who uses the system to manage ingredients in the refrigerator and receive recipe suggestions.

[0567] "Means for acquiring images of the contents of the refrigerator" is a function that allows a user to take a photo of the food in the refrigerator using a smartphone or tablet and send the image data to the system.

[0568] "Image recognition means" refers to technology that analyzes acquired image data and identifies ingredients in the image. A deep learning-based algorithm is used.

[0569] "Food items" refers to ingredients or food stored in the refrigerator, such as tomatoes, chicken, and broccoli.

[0570] The "means for suggesting recipes" is a function that provides optimal cooking methods and cooking ideas based on information about the identified food items.

[0571] "Detailed information" refers to additional information about the suggested recipe, such as specific cooking instructions, a list of ingredients needed, calorie information, and nutritional balance.

[0572] "Feedback means" refers to a function that allows users to send ratings and comments on suggested recipes to the system.

[0573] "Means for accumulating feedback and improving the recipe suggestion algorithm" refers to a method for accumulating feedback data provided by users in a database and analyzing that data to improve the accuracy of the recipe suggestion algorithm.

[0574] "User's past choices and preferences" refers to the recipes and ingredients the user has chosen in the past, the types of dishes they like, and other information that is taken into consideration when suggesting recipes.

[0575] As an embodiment of the present invention, we provide a system in which a user scans the contents of their refrigerator using a smartphone or tablet, identifies food items using image recognition technology, and suggests optimal recipes based on that information. This system has the user, a terminal, and a server as its main elements, and these elements function in cooperation with each other.

[0576] User scans the contents of the refrigerator

[0577] The user launches the app installed on their smartphone. Using the app's camera function, the user takes a picture of the contents of the refrigerator. For example, imagine a scenario in which the user takes out "tomatoes," "chicken," and "broccoli" from the refrigerator and takes a picture of them. The smartphone acquires the captured image and sends it to the server via the HTTPS protocol.

[0578] The server performs image recognition

[0579] The server receives the image of the refrigerator interior sent by the user. The server is equipped with a deep learning-based image recognition module (e.g., TensorFlow, PyTorch) that recognizes food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. Information about the recognized food items is stored in a database (e.g., MySQL, MongoDB).

[0580] Server suggests recipes

[0581] The server searches a recipe database based on the recognized food item list. The recipe database contains a vast amount of recipe information, and searches for and suggests recipes related to specific food items. The server also takes into account the user's past selection history and preferences to filter and prioritize the search results. For example, recipes such as "grilled chicken and broccoli" and "pasta with tomato and chicken" are suggested.

[0582] The device notifies the user of recipe suggestions

[0583] The server sends the generated recipe list to the user's device. The smartphone app displays the suggested recipe list on the user's screen and notifies the user with an interactive message:

[0584] We recommend the following recipes for dinner tonight:

[0585] 1. Grilled Chicken and Broccoli

[0586] 2. Tomato and chicken pasta

[0587] Which one would you like?

[0588] User selects a recipe and receives details

[0589] The user selects one of the suggested recipes. For example, if the user selects "grilled chicken and broccoli," the smartphone sends the selection to the server. The server generates detailed information about the selected recipe and sends it to the device. The detailed information includes cooking instructions, a list of ingredients, calorie information, and nutritional balance. The device displays this information on the user's screen.

[0590] Users provide feedback

[0591] After trying the suggested recipes, users can provide feedback from within the app, such as "It was delicious" or "I wish it was a little saltier." The device then sends this feedback information to the server.

[0592] The server collects feedback and improves the algorithm

[0593] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[0594] Example prompts for generative AI models

[0595] Prompt statement:

[0596] "Design a system that analyzes a user's image of the inside of a refrigerator, identifies the food items contained within, and suggests optimal recipes based on that. Include recipe filtering that takes into account the user's past choices and preferences."

[0597] In this way, the system efficiently utilizes the contents of the refrigerator and significantly reduces the effort required for users to make dinner. For example, users can select a recipe based on the items in the refrigerator and then recreate it at home, saving time and effort.

[0598] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0599] Step 1:

[0600] A user scans the contents of the refrigerator with their smartphone

[0601] Input: App installed on smartphone, user's shooting operation

[0602] How it works: The user launches the app and uses the camera to take a picture of the food in the refrigerator. They can also take multiple images in succession.

[0603] Output: Image data of the inside of the refrigerator

[0604] Step 2:

[0605] The device sends the image to the server

[0606] Input: Captured image, smartphone network connection

[0607] What it does: The app sends the acquired image data to a server using the HTTPS protocol.

[0608] Output: Image data sent to the server

[0609] Step 3:

[0610] The server receives and stores the image

[0611] Input: Submitted image data

[0612] What it does: The server saves the image data in a temporary directory, specifically as a file, for later processing.

[0613] Output: Saved image file

[0614] Step 4:

[0615] The server performs image recognition

[0616] Input: Saved image file

[0617] How it works: The server's deep learning-based image recognition module analyzes the image and identifies the food item, using a Convolutional Neural Network (CNN) model (e.g., TensorFlow or PyTorch).

[0618] Output: A list of identified food items (e.g., "tomato," "chicken," "broccoli")

[0619] Step 5:

[0620] The server stores the food item list in a database.

[0621] Input: A list of identified food items

[0622] What it does: Stores the identified food item information in JSON format in a database (e.g. MySQL, MongoDB).

[0623] Output: Stored food item data

[0624] Step 6:

[0625] The server searches for and suggests recipes

[0626] Input: Stored food item data

[0627] How it works: Searches for recipes related to a specified food item in a database, filters the search results based on past history and preferences, and extracts relevant recipes using SQL queries.

[0628] Output: A list of suggested recipes (e.g. "Grilled Chicken and Broccoli" or "Pasta with Tomato and Chicken")

[0629] Step 7:

[0630] The device notifies the user of the recipe list

[0631] Input: Suggested recipe list

[0632] Operation: The smartphone app displays the recipe list received from the server in the form of a message to the user.

[0633] Output: Recipe selection message screen

[0634] Step 8:

[0635] The user selects a recipe

[0636] Input: User selection of recipe

[0637] How it works: The user selects one of the suggested recipes on the screen, and the selection is sent to the server.

[0638] Output: The ID of the selected recipe.

[0639] Step 9:

[0640] The server generates and sends the recipe details.

[0641] Input: ID of the selected recipe

[0642] What it does: Retrieves detailed information about the selected recipe from the recipe database (e.g., cooking instructions, ingredient list, calorie information, etc.) and sends the information in JSON format to the smartphone.

[0643] Output: Recipe details

[0644] Step 10:

[0645] The device displays detailed information to the user

[0646] Input: Recipe details received from the server

[0647] What it does: The app parses the details and displays them on the screen in a user-friendly format.

[0648] Output: Recipe details screen displayed

[0649] Step 11:

[0650] Users provide feedback

[0651] Input: User-initiated feedback input

[0652] How it works: The user enters feedback about a recipe and the app sends that data to the server.

[0653] Output: Feedback data sent

[0654] Step 12:

[0655] The server collects feedback and improves the algorithm

[0656] Input: Received feedback data

[0657] How it works: The server stores the feedback data in a database, analyzes it, and improves the recipe suggestion algorithm. It also retrains the machine learning model to reflect this in future recipe suggestions.

[0658] Output: An improved recipe suggestion algorithm

[0659] (Application example 1)

[0660] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0661] In recent years, food waste and shortening cooking times at home have become social issues. There is a demand for efficient use of refrigerator contents and reduction of waste, but current technology requires the time-consuming manual management of food in the refrigerator, making it difficult to find appropriate recipes based on that information. Furthermore, if necessary ingredients are in short supply, users have to go shopping, which is a significant burden on them. There is a need for a system that can solve these issues.

[0662] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0663] In this invention, the server includes means for a user to acquire an image of the contents of the refrigerator, image recognition means for identifying the acquired image and specifying food items, means for suggesting recipes based on information about the identified food items, means for providing detailed information about the suggested recipes, and means for automatically listing missing ingredients and ordering the ingredients from a delivery service, thereby enabling efficient use of the contents of the refrigerator and saving time and effort.

[0664] A "user" is an individual or group that uses this system to manage the contents of their refrigerator and receive recipe suggestions.

[0665] A "refrigerator" is a household or commercial appliance used to store food at the appropriate temperature.

[0666] "Images" are photos of food in the refrigerator taken by the user using a smartphone or tablet.

[0667] "Acquisition" means that a user takes a picture of the inside of the refrigerator using a smartphone or tablet and imports that image data into the system.

[0668] "Ingredient items" are specific foods and ingredients stored in the refrigerator.

[0669] "Image recognition" is a technology that analyzes captured images and identifies the food items contained within them.

[0670] "Recipe suggestion" refers to providing a cooking method suitable for the user based on information about the identified ingredient items.

[0671] "Detailed information" refers to information about the proposed recipe, including specific cooking steps, necessary ingredients, calorie information, and the like.

[0672] A "missing ingredient" is an ingredient item that is needed to create a suggested recipe but is not present in the refrigerator.

[0673] "Delivery service" is a service that allows users to order missing materials online and have these materials delivered promptly.

[0674] "Automatic Listing" is a function that automatically generates a list of ingredients that are in short supply based on recipe suggestions.

[0675] An "order" is a series of actions and procedures in which a user requests a delivery service to provide missing materials and arranges for delivery.

[0676] To implement this invention, a user uses a smartphone to scan the contents of their refrigerator, and the server uses image recognition technology to identify food items and suggests optimal recipes based on that information. Furthermore, the system automatically lists any missing ingredients and allows the user to order those ingredients via a delivery service.

[0677] Hardware and Software Configuration

[0678] Hardware:

[0679] On the user's phone:

[0680] Use the camera feature to scan the contents of your refrigerator.

[0681] software:

[0682] server:

[0683] Use Deep Learning-based frameworks such as TensorFlow / Keras for image recognition.

[0684] PIL (Python Imaging Library) is used for image processing.

[0685] The requests library is used to process HTTP requests.

[0686] Terminal app:

[0687] The contents of the refrigerator are scanned and the images are sent to a server.

[0688] The suggested recipe and the list of ingredients that are in short supply received from the server are displayed to the user.

[0689] Call the delivery service API to order the required ingredients.

[0690] System Operation

[0691] Fridge scanning and image recognition

[0692] The user launches the "Refrigerator-Linked Delivery Assistant" app on their smartphone and uses the camera to take a picture of the food in their refrigerator. The image data is sent from the device app to the server. The server uses a deep learning-based image recognition system to identify the food items in the image and generate a list of identified items. For example, "tomatoes," "chicken," and "onions" in the refrigerator are recognized.

[0693] Recipe suggestions and a list of missing ingredients

[0694] Based on the recognized food item list, the server searches a recipe database and suggests several recipes suitable for the user, taking into account the user's past selection history and preferences. The server also automatically lists any ingredients needed for the suggested recipes that are missing from the refrigerator. For example, a recipe for "Tomato Chicken Salad" may be suggested, but the server will point out that the required "lettuce" is missing.

[0695] Automating delivery orders

[0696] The list of missing ingredients is sent from the device app to the delivery service API, which automatically places an order, allowing users to get the missing ingredients immediately. Once the delivery order is successfully processed, the order status is displayed on the device.

[0697] Specific examples

[0698] For example, suppose a user scans the "tomatoes," "chicken," and "onions" in their refrigerator. Based on this, the server suggests a recipe for "Tomato Chicken Salad" and lists the missing ingredient, "lettuce," which is needed but not present. The server then automatically orders "lettuce" using a delivery service API, and the user can receive the ingredients via delivery.

[0699] Prompt Sentence Examples

[0700] text

[0701] Scan the image of the refrigerator

[0702] recipes, missing_items = scan_fridge('fridge_image.jpg')

[0703] Create a delivery order

[0704] if missing_items:

[0705] order_status = order_missing_items(missing_items)

[0706] print("Order Status:", order_status)

[0707] View suggested recipes

[0708] if recipes:

[0709] for idx, recipe in enumerate(recipes):

[0710] print(f"Recipe {idx + 1}:{recipe['name']}")

[0711] print(f"ingredients:{recipe['ingredients']}")

[0712] print(f"Steps:{recipe['steps']}")

[0713] print("\n")

[0714] In this way, by efficiently managing food items in the refrigerator and easily obtaining the necessary ingredients, users can significantly save time and effort in cooking.

[0715] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0716] Step 1:

[0717] The user launches the "Refrigerator-Linked Delivery Assistant" app on their smartphone and uses the camera to take a picture of the inside of the refrigerator. The image is then temporarily saved in the smartphone's local storage.

[0718] Input: Image of the inside of the refrigerator taken with a smartphone camera

[0719] Output: Images saved in the smartphone's local storage

[0720] Specific operation: The user launches the app, takes a picture of the inside of the refrigerator with their smartphone camera, and saves the image.

[0721] Step 2:

[0722] The device app sends the saved image to the server, which receives it as an HTTP request.

[0723] Input: A picture of a refrigerator stored on a smartphone

[0724] Output: Image data sent to the server

[0725] Specific operation: The terminal application obtains the image data and sends the image to the server via an HTTP request.

[0726] Step 3:

[0727] The server inputs the received image data into a deep learning-based image recognition system to identify food items, and the list of identified items is stored in a database on the server.

[0728] Input: Image data sent to the server

[0729] Output: A list of identified food items

[0730] What it does: The server uses TensorFlow / Keras to analyze the image and identify the food items in it, resulting in a list of items.

[0731] Step 4:

[0732] The server searches a recipe database based on the recognized food item list and suggests multiple recipes suitable for the user. The suggested recipes are generated as a list, taking into account the user's past selection history and preferences.

[0733] Input: List of identified food items

[0734] Output: A list of suggested recipes

[0735] What it does: The server runs a database query to find relevant recipes and prioritizes them based on the user's historical data.

[0736] Step 5:

[0737] The server creates a list of missing ingredients based on the proposed recipe and sends it to the device app, which then passes the list to the delivery service API.

[0738] Input: Suggested recipe list

[0739] Output: List of missing materials

[0740] What it does: The server matches the suggested recipe with the current food items, lists any missing ingredients, and sends them to the delivery service API.

[0741] Step 6:

[0742] The terminal app displays the list of missing ingredients received from the server to the user, automatically places a delivery service order, and notifies the user of the order status of the missing ingredients.

[0743] Input: List of materials in short supply received from the server

[0744] Output: The list displayed to the user and the order status

[0745] What it does: The device app displays a list of missing ingredients and updates the user on the status of their delivery order in real time.

[0746] Step 7:

[0747] The user selects a suggested recipe and receives detailed information about it on their device, including cooking instructions, ingredients needed, and calorie information.

[0748] Input: Suggested recipe list

[0749] Output: Detailed information about the selected recipe

[0750] Specific behavior: The user selects a suggested recipe within the app and receives detailed information from the server.

[0751] Step 8:

[0752] After cooking, users provide feedback through the device app, which is then sent to the server and used to suggest recipes for future meals.

[0753] Input: User feedback

[0754] Output: Feedback data stored on the server

[0755] How it works: The user enters feedback into the device app and sends it to the server, which stores it in a database and uses it to improve the accuracy of the algorithm.

[0756] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0757] This invention provides a system that allows a user to scan the contents of their refrigerator using a smartphone or tablet, and then combines image recognition technology with emotion recognition technology to identify food items and suggest optimal recipes based on the user's emotional state. Below, the program's processing is explained in natural language using concrete examples.

[0758] User scans the contents of the refrigerator

[0759] The user launches the "Dinner Omakase" app installed on their smartphone. They then use the app's camera function to take a picture of the contents of their refrigerator. The device then acquires the captured image data (image of the inside of the refrigerator) and sends this image to the server.

[0760] The server performs image recognition

[0761] The server receives the image of the refrigerator interior sent by the user. A deep learning-based image recognition module is installed on the server, and this module is used to recognize food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. The list of identified food items is stored in a database.

[0762] The server performs emotion recognition.

[0763] When a user is using a device with a camera, the device captures the user's facial expression data and sends the data to an emotion recognition engine in the server. The emotion recognition engine analyzes the facial expression data and identifies the user's emotional state (e.g., "satisfied," "fatigued," "stressed," etc.).

[0764] Server suggests recipes

[0765] The server searches a recipe database based on the identified food item list and emotion recognition results to retrieve relevant recipes. The recipe database contains a vast amount of recipe information, and searches for and suggests recipes that are suitable for the identified food items and the user's emotional state. For example, if the user is in a "fatigue" state, recipes that are easy to make and can be made in a short time are prioritized.

[0766] The device notifies the user of recipe suggestions

[0767] The server generates a list of suggested recipes and sends it to the user's device. The device's app displays the list of suggested recipes on the user's screen and sends the following interactive message:

[0768] We recommend the following recipes for dinner tonight:

[0769] 1. Grilled Chicken and Broccoli (Cooking time: 20 minutes)

[0770] 2. Tomato and Chicken Pasta (Cooking time: 15 minutes)

[0771] Which one would you like?

[0772] The user selects one of the suggested recipes.

[0773] User selects a recipe and receives details

[0774] The device receives the recipe selected by the user and sends that information to the server. The server generates detailed information about the selected recipe and sends it to the device. The detailed information includes cooking instructions, a list of ingredients, calorie information, and nutritional balance. The device displays this information on the user's screen.

[0775] Users provide feedback

[0776] After trying the suggested recipes, users can provide feedback from within the app, such as "It was delicious" or "I wish it was a little saltier." The device then sends this feedback information to the server.

[0777] The server collects feedback and improves the algorithm

[0778] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[0779] In this way, by combining emotion recognition technology, the system can quickly suggest optimal recipes based on the user's current emotional state, significantly reducing the effort and stress of cooking dinner. As a specific example, a user can select a recipe based on the items in the refrigerator and their own emotional state, and then actually recreate it at home, saving time and effort.

[0780] The processing flow will be explained below.

[0781] Step 1:

[0782] The user launches the "Dinner Omakase" app on their smartphone and uses the app's camera function to take a picture of the contents of the refrigerator. The device temporarily saves the image as a "refrigerator image."

[0783] Step 2:

[0784] The terminal sends the "refrigerator interior image" to the server, and the image data is transferred to the server via the network.

[0785] Step 3:

[0786] The server receives the "refrigerator image" sent by the user. The received image data is passed to the image recognition module in the server.

[0787] Step 4:

[0788] The server's image recognition module uses deep learning-based algorithms to analyze and identify food items in the image, such as "tomato," "chicken," and "broccoli."

[0789] Step 5:

[0790] The server generates a list of the identified food items and stores it in a database. The identified food item information is stored in the database.

[0791] Step 6:

[0792] If the user is using a device with a camera, the device captures the user's facial expression data and sends the data to an emotion recognition engine.

[0793] Step 7:

[0794] The server's emotion recognition engine analyzes the facial expression data and identifies the user's emotional state (e.g., "satisfied," "fatigued," "stressed," etc.) This emotional state information is stored in a database.

[0795] Step 8:

[0796] The server searches the recipe database based on the food item list and emotion recognition results to retrieve related recipes. Multiple recipes are found in the recipe database.

[0797] Step 9:

[0798] The server prioritizes the retrieved recipes based on the user's past selection history, preferences, and current emotional state. For example, if the user is "fatigued," recipes that are easy to prepare and can be made quickly will be prioritized.

[0799] Step 10:

[0800] The server generates a prioritized recipe list and transmits it to the user's terminal, where it is displayed in a format that the user can view.

[0801] Step 11:

[0802] The terminal will display a message to the user similar to:

[0803] We recommend the following recipes for dinner tonight:

[0804] 1. Grilled Chicken and Broccoli (Cooking time: 20 minutes)

[0805] 2. Tomato and Chicken Pasta (Cooking time: 15 minutes)

[0806] Which one would you like?

[0807] The user selects one of the suggested recipes.

[0808] Step 12:

[0809] The terminal receives the recipe selected by the user and transmits the information to the server.

[0810] Step 13:

[0811] The server generates detailed information about the selected recipe and sends it to the device, including cooking instructions, a list of ingredients, calorie information, and nutritional balance.

[0812] Step 14:

[0813] The device displays the detailed information received from the server to the user, allowing the user to check the specific cooking steps and ingredient information for the selected recipe.

[0814] Step 15:

[0815] After the user tries the recipe, they submit feedback from within the app. The device collects the user's feedback information and sends it to the server.

[0816] Step 16:

[0817] The server stores the received feedback in a database, analyzes the stored feedback information, and updates the recommendation algorithm to improve the accuracy of future recipe suggestions.

[0818] Example 2

[0819] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0820] Conventional food management and recipe recommendation systems have difficulty not only accurately identifying the food in a user's refrigerator, but also automating optimal recipe recommendations that take into account the user's emotional state. This has left users unable to quickly choose an appropriate dish that suits their current mood while utilizing the food in their refrigerator, often resulting in stress when preparing dinner. To solve this problem, a recipe recommendation system that simultaneously considers the food in the refrigerator and the user's emotional state is needed.

[0821] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0822] In this invention, the server includes means for allowing a user to acquire an image of the contents of the refrigerator, image recognition means for identifying the acquired image and specifying food items, emotion recognition means for acquiring data on the user's facial expression and analyzing the data to identify the user's emotional state, means for suggesting recipes based on information on the identified food items and the user's emotional state, and means for providing detailed information on the suggested recipes. This makes it possible to quickly suggest optimal recipes based on information on the food items in the refrigerator and the user's current emotional state.

[0823] "User" refers to an individual who uses the system to scan the contents of their refrigerator and receive recipe suggestions.

[0824] "Means of acquiring images" refers to the process in which a user takes a photo of the food in the refrigerator using the camera function of a smartphone or other device.

[0825] "Image recognition means" refers to technology for analyzing acquired image data and identifying food items in the refrigerator.

[0826] "Facial expression data" refers to image data captured by a camera of a user's face, and is used for emotion recognition.

[0827] "Emotion recognition means" refers to technology that analyzes facial expression data and identifies the user's emotional state (e.g., "satisfaction," "fatigue," "stress," etc.).

[0828] "Means for suggesting recipes" refers to a function that selects and suggests appropriate recipes based on information about the identified food items and the user's emotional state.

[0829] "Means for providing detailed information" refers to a function that provides users with detailed information such as cooking instructions for the suggested recipe, a list of required ingredients, calorie information, and nutritional balance.

[0830] The present invention is a system that allows a user to scan the contents of a refrigerator using a smartphone or tablet, and combines image recognition technology and emotion recognition technology to identify food items and suggest optimal recipes based on the user's emotional state. Next, a specific embodiment of this system will be described.

[0831] The system uses the following hardware and software:

[0832] Mobile devices such as smartphones and tablets

[0833] Camera on the device

[0834] Deep learning-based image recognition modules (e.g., TensorFlow and PyTorch)

[0835] Emotion recognition engine (e.g., Microsoft Azure Face API)

[0836] Recipe Database

[0837] Specific steps

[0838] User scans the contents of the refrigerator

[0839] The user launches the "Dinner Omakase" app installed on their smartphone and uses the camera to take a picture of the contents of the refrigerator. The device acquires the captured image data and sends it to the server in real time.

[0840] The server performs image recognition

[0841] The server receives the image of the refrigerator sent by the user and uses a deep learning-based image recognition module to recognize food items in the image, such as "tomato," "chicken," and "broccoli," and stores this information in a database.

[0842] The server performs emotion recognition.

[0843] When a user is using a device with a camera, the device captures the user's facial expression data and sends it to an emotion recognition engine in the server. The server analyzes the facial expression data and identifies the user's emotional state (e.g., "satisfied," "fatigued," "stressed," etc.).

[0844] Server suggests recipes

[0845] The server searches the recipe database based on the identified food item list and the emotion recognition results to retrieve relevant recipes. For example, if the user is in a "fatigue" state, it will prioritize recipes that are easy to make and can be made in a short time.

[0846] The device notifies the user of recipe suggestions

[0847] The server generates a list of suggested recipes and sends it to the user's device, where the app displays the list on the screen and provides specific lists and options.

[0848] User selects a recipe and receives more information

[0849] The user selects one of the suggested recipes. The selected recipe information is sent from the device to the server, and the server generates detailed cooking instructions, a list of ingredients, calorie information, nutritional balance, and other information and sends it to the device. The device then displays this information on the user's screen.

[0850] Users provide feedback

[0851] After trying the suggested recipes, users provide feedback from within the app, which the device then captures and sends to the server.

[0852] The server accumulates feedback and improves the algorithm

[0853] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[0854] Examples of prompt statements

[0855] For example, if a user uses a "Dinner Picker" app to scan the items "clams" and "garlic" in the refrigerator, and the app recognizes the user's facial expression as indicating a "fatigue" state, the prompt sentence based on this example might be:

[0856] Use the "Dinner Omakase" app to scan the contents of your refrigerator (clams, garlic) and suggest the best dinner recipe through image and emotion recognition. The user's emotional state is "fatigue."

[0857] In this way, by combining emotion recognition technology, the system can quickly suggest optimal recipes based on the user's current emotional state, significantly reducing the effort and stress of cooking dinner.

[0858] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0859] Processing flow

[0860] Step 1:

[0861] The user launches the app installed on their smartphone and takes a picture of the contents of the refrigerator. At this time, the user uses the camera function in the app to capture image data (images of the food in the refrigerator). The image data obtained as input is saved in the device's internal storage. Specifically, the user taps the camera icon on the app screen and takes a picture of the inside of the refrigerator with the smartphone's camera.

[0862] Step 2:

[0863] The device sends the acquired image data to the server. At this time, the device's app uploads the image data to the server via an Internet connection. The input is the image data acquired in step 1, and the output is the image data sent to the server. Specifically, the device uploads the data to the server using a background process.

[0864] Step 3:

[0865] The server receives image data sent by the user and identifies food items using an image recognition module. The image data received as input is analyzed using a deep learning-based image recognition algorithm (e.g., TensorFlow or PyTorch) to generate a list of food items. Specifically, the server calls the image recognition API and generates a list of items such as "tomato," "chicken," and "broccoli."

[0866] Step 4:

[0867] The server stores the identified food item list in a database, where the input is the food item list obtained in step 3 and the output is the information stored in the database. Specifically, the server opens a database connection and executes an SQL query to store the listed food items.

[0868] Step 5:

[0869] When a user is using a device with a camera, the device acquires the user's facial expression data and sends it to the emotion recognition engine. The input is the user's facial expression image data, and the output is the transmission of facial expression data from the device to the server. Specifically, the user shows an expression in front of the camera, and the device captures it and uploads it to the server.

[0870] Step 6:

[0871] The server analyzes the received facial expression data using an emotion recognition engine to identify the user's emotional state. The input is facial expression data, and the output is the identified emotional state (e.g., "satisfied," "fatigued," "stressed," etc.). Specifically, the server calls an emotion recognition API (e.g., Microsoft Azure Face API) to analyze the emotional state.

[0872] Step 7:

[0873] The server searches the recipe database based on the identified food item list and emotion recognition results to suggest optimal recipes. The input is the food item list and the emotional state, and the output is a list of related recipes. Specifically, the server generates a query and executes an SQL query to retrieve suitable recipes from the recipe database.

[0874] Step 8:

[0875] The server generates a list of suggested recipes and sends it to the user's device. The input is the recipe list, and the output is sending the recipe list to the device. Specifically, the server sends the generated recipe list to the device via push notification.

[0876] Step 9:

[0877] The device displays the suggested recipe list on the user's screen. The input is the recipe list sent from the server, and the output is the list displayed on the user's screen. In concrete terms, the app receives the recipe list and displays it using a UI component.

[0878] Step 10:

[0879] The user selects one of the suggested recipes. The input is the displayed recipe list, and the output is the selected recipe information. Specifically, the user selects a recipe by tapping on the screen.

[0880] Step 11:

[0881] The terminal sends the selected recipe information to the server. The input is the recipe information selected by the user, and the output is the sending of a recipe list to the server. In concrete terms, the terminal uploads the selected content to the server.

[0882] Step 12:

[0883] The server generates detailed information for the selected recipe and sends it to the terminal. The input is the selected recipe information, and the output is the detailed recipe information. Specifically, the server generates detailed cooking instructions and ingredient lists and sends them to the terminal.

[0884] Step 13:

[0885] The device displays detailed recipe information on the user's screen. The input is the detailed recipe information sent from the server, and the output is the detailed information displayed on the user's screen. Specifically, the app receives the detailed information and displays it as a description or list.

[0886] Step 14:

[0887] The user enters feedback within the app. The input is the user's feedback comment, and the output is saving the feedback information to the device. Specifically, the user writes a comment in the feedback form and taps the submit button.

[0888] Step 15:

[0889] The terminal sends feedback information to the server. The input is the feedback information provided by the user, and the output is the transmission of the feedback information to the server. In concrete terms, the terminal uploads the transmitted feedback to the server.

[0890] Step 16:

[0891] The server accumulates the feedback sent by the user in a database and reflects it in future recipe suggestions. The input is the feedback data, and the output is an updated recipe suggestion algorithm. Specifically, the server stores the feedback data in storage and uses it to improve the machine learning algorithm.

[0892] (Application example 2)

[0893] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0894] Conventional recipe suggestion systems could suggest recipes based on the contents of a user's refrigerator, but they could not take the user's emotional state into account. This made it difficult for users to find products and recipes that matched their emotions at any given time. The present invention aims to solve these problems by providing a system that detects a user's emotional state and suggests optimal products and recipes based on that emotional state.

[0895] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0896] In this invention, the server includes means for a user to acquire an image of the contents of the refrigerator, image recognition means for identifying food items by recognizing the acquired image, means for suggesting recipes based on information on the identified food items, means for providing detailed information on the suggested recipes, emotion recognition means for scanning the user's facial expression to identify their emotional state, means for recommending products based on the user's emotional state, and means for providing detailed information on the recommended products, thereby enabling the user to receive recommendations that are optimal for their emotional state at the time.

[0897] "User" refers to an individual who uses the system.

[0898] "Means for acquiring an image of the contents of the refrigerator" refers to a function that allows a user to take a picture of the contents of the refrigerator using the camera on a smartphone or tablet and acquire the image data.

[0899] "Image recognition means" refers to technology for analyzing captured image data and identifying food items present in the image.

[0900] "Food items" refer to ingredients or foods present in an image, including specific items such as tomatoes or chicken.

[0901] The "means for suggesting recipes" refers to a function that provides the user with appropriate cooking methods and menus based on information about food items identified by the image recognition means.

[0902] "Means for providing detailed information" refers to a function that presents the user with specific cooking steps, a list of ingredients, calorie information, etc. for a recipe.

[0903] "Emotion recognition means for scanning facial expressions to identify emotional states" refers to technology for analyzing a user's facial expression data and identifying their emotional state at that time (e.g., joy, fatigue, etc.).

[0904] "Means for recommending products" refers to a function that suggests appropriate products to users based on their emotional state and past behavioral history.

[0905] "Means for providing detailed information about recommended products" refers to a function that displays detailed information about recommended products, such as descriptions, prices, and reviews, to users.

[0906] The present invention relates to a system that allows users to scan the contents of their refrigerator using a smartphone or tablet, combines image recognition technology with emotion recognition technology to identify food items, and suggests optimal recipes and products based on the user's emotional state.

[0907] System configuration

[0908] The system consists of the following main components:

[0909] 1. User device: A device equipped with a camera and display, such as a smartphone or tablet.

[0910] 2. Server: A cloud service that performs image recognition, emotion recognition, and recipe and product suggestions.

[0911] 3. Database: Stores food items, emotional states, recipes, product information, user behavior history and feedback.

[0912] Program execution procedure

[0913] 1. User scans the contents of the refrigerator

[0914] The user launches the dedicated application and takes a picture of the contents of the refrigerator using the camera function of their smartphone or tablet. The device acquires the image data and sends it to the server.

[0915] 2. Image recognition by the server

[0916] The server receives the image of the refrigerator sent by the user. The server is equipped with a deep learning-based image recognition module, which recognizes food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. The list of identified food items is stored in a database.

[0917] 3. Emotion recognition

[0918] When a user uses a device with a camera, the device captures the user's facial expression data and sends it to the server. The emotion recognition engine in the server analyzes the facial expression data and identifies the user's emotional state. For example, emotional states such as "joy," "fatigue," and "stress" are analyzed.

[0919] 4. Recipe and product suggestions

[0920] The server searches the recipe and product database based on the identified food item list and emotion recognition results to retrieve related recipes and products. Related recipes and products are suggested based on the user's emotional state. For example, if the user is "fatigued," recipes that are easy to make in a short time and products that help them relax are suggested first.

[0921] 5. Notifications on user devices

[0922] The server generates suggested recipes and product lists and sends them to the user's device. The device app displays the suggested recipes and product lists on the user's screen and sends interactive messages. For example, the following message might be displayed:

[0923] Here are some products we recommend for you right now:

[0924] 1. Grilled Chicken and Broccoli (Cooking time: 20 minutes)

[0925] 2. Tomato and Chicken Pasta (Cooking time: 15 minutes)

[0926] 3. Stress Relief Ball

[0927] 4. Herbal tea

[0928] Which one would you like?

[0929] Hardware and software used

[0930] Hardware:

[0931] Smartphone or tablet camera: Used to capture images of the user's facial expressions and the inside of the refrigerator.

[0932] software:

[0933] Python: The implementation language for the entire program.

[0934] opencv-python: Used to process facial expression and fridge image data.

[0935] Pillow: A library for image processing.

[0936] numpy: Used to process data.

[0937] TensorFlow: Emotion recognition using pre-trained models.

[0938] Prompt Sentence Examples

[0939] Write code to analyze a user's emotional state using facial images and recommend products that best match their emotions. Use the camera function to capture facial images and use TensorFlow to recognize emotions with a pre-trained model. Write a set of Python code to display a list of products based on the recognized emotion.

[0940] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0941] Step 1:

[0942] The user launches an application on their smartphone or tablet and uses the camera function to take a picture of the contents of the refrigerator. The device acquires the captured image data (image of the inside of the refrigerator) and sends this image to the server. The input is the image data captured by the user, and the output is the transfer of the image data to the server.

[0943] Step 2:

[0944] The server receives an image of the inside of the refrigerator sent by the user. It uses a deep learning-based image recognition module to recognize food items in the image. The input is image data of the inside of the refrigerator, and the output is a list of identified food items. Specifically, the image data is preprocessed and input into a model to identify food items.

[0945] Step 3:

[0946] The identified food item list is saved in a database. The input is the food item list output from image recognition, and the output is the process of saving it to the database. Specifically, the type and quantity of food items are written to the database.

[0947] Step 4:

[0948] The user takes a photo of their own facial expression using the camera function of their smartphone or tablet. The device acquires the facial expression data and sends it to the server. The input is the facial expression image taken by the user, and the output is the transfer of the facial expression data to the server.

[0949] Step 5:

[0950] The server receives the facial expression images and analyzes the facial expression data using an emotion recognition engine. The input is the facial expression image data, and the output is the user's emotional state. Specifically, the facial expression data is preprocessed and input into an emotion recognition model to identify the emotional state.

[0951] Step 6:

[0952] The server searches the recipe and product databases based on the identified food item list and emotional state to retrieve related recipes and products. The input is the food item list and emotional state, and the output is a list of recommended recipes and products. Specifically, it extracts information that matches the conditions from the recipe database and product database.

[0953] Step 7:

[0954] The server generates a list of recommended recipes and products and sends it to the user's device. The input is the recommendation list, and the output is data transmission to the user's device. Specifically, the server formats the selected information and delivers it to the user's device.

[0955] Step 8:

[0956] The user device displays the suggested recipes and product list on the screen and presents interactive messages to the user. For example, it presents multiple options along with the message "Here are the products we recommend for you right now:" The input is the recommendation list data from the server, and the output is the display to the user. Specifically, the information is presented visually using a GUI.

[0957] Step 9:

[0958] The user checks the detailed information of the suggested recipes and products and selects a specific recipe or product. The input is the user's selection, and the output is the transmission of the selected data to the server. Specifically, the system detects the user's action of pressing the selection button and sends the selected information to the server.

[0959] Step 10:

[0960] The server receives the user's selection information, generates detailed information about the selected recipe or product, and sends it to the user's device. The input is the user's selection information, and the output is a response with detailed information. Specifically, the server retrieves detailed information from the database, formats it, and provides it to the user.

[0961] Step 11:

[0962] The user terminal displays detailed information on the screen and provides cooking instructions and product details. The input is the detailed information data sent from the server, and the output is the display to the user. Specifically, the detailed information is presented visually using a GUI.

[0963] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0964] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0965] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0966] [Third embodiment]

[0967] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0968] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0969] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0970] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0971] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0972] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0973] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0974] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0975] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0976] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0977] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0978] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0979] As an embodiment of this invention, we provide a system that allows a user to scan the contents of their refrigerator using a smartphone or tablet, identifies food items using image recognition technology, and suggests optimal recipes based on that information. Next, we will explain the program's processing in natural language using concrete examples.

[0980] User scans the contents of the refrigerator

[0981] The user launches the "Dinner Omakase" app installed on their smartphone. Using the app's camera function, the user takes a picture of the contents of the refrigerator. The smartphone acquires the image and sends it to the server.

[0982] The server performs image recognition

[0983] The server receives the image of the refrigerator interior sent by the user. A deep learning-based image recognition module is installed on the server, and this module is used to recognize food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. The list of identified food items is stored in a database.

[0984] Server suggests recipes

[0985] The server searches a recipe database based on the identified food item list. The recipe database contains a vast amount of recipe information, and searches for and suggests recipes related to the specific food items. The server then filters and prioritizes the search results, taking into account the user's past selection history and preferences. For example, recipes such as "grilled chicken and broccoli" and "pasta with tomato and chicken" are suggested.

[0986] The device notifies the user of recipe suggestions

[0987] The server generates a list of suggested recipes and sends it to the user's device. The device's app displays the list of suggested recipes on the user's screen and sends the following interactive message:

[0988] We recommend the following recipes for dinner tonight:

[0989] 1. Grilled Chicken and Broccoli

[0990] 2. Tomato and chicken pasta

[0991] Which one would you like?

[0992] User selects a recipe and receives details

[0993] The user selects one of the suggested recipes. For example, if the user selects "grilled chicken and broccoli," the device sends the selection to the server. The server generates detailed information about the selected recipe and sends it to the device. The detailed information includes cooking instructions, a list of ingredients, calorie information, and nutritional balance. The device displays this information on the user's screen.

[0994] Users provide feedback

[0995] After trying the suggested recipes, users can provide feedback from within the app, such as "It was delicious" or "I wish it was a little saltier." The device then sends this feedback information to the server.

[0996] The server collects feedback and improves the algorithm

[0997] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[0998] As described above, this system efficiently utilizes the contents of the refrigerator and significantly reduces the effort required for users to make dinner. As a specific example, the user can select a recipe based on the items in the refrigerator and then recreate it at home, saving time and effort.

[0999] The processing flow will be explained below.

[1000] Step 1:

[1001] The user launches the "Dinner Omakase" app on their smartphone and uses the app's camera function to take a picture of the contents of the refrigerator. The device temporarily saves the image as a "refrigerator image."

[1002] Step 2:

[1003] The terminal sends the "refrigerator interior image" to the server, and the image data is transferred to the server via the network.

[1004] Step 3:

[1005] The server receives the "refrigerator image" sent by the user. The received image data is passed to the image recognition module in the server.

[1006] Step 4:

[1007] The server's image recognition module uses deep learning-based algorithms to analyze and identify food items in the image, such as "tomato," "chicken," and "broccoli."

[1008] Step 5:

[1009] The server generates a list of the identified food items and stores it in a database. The identified food item information is stored in the database.

[1010] Step 6:

[1011] The server searches the recipe database based on the food item list to retrieve related recipes. Multiple recipes are found in the recipe database.

[1012] Step 7:

[1013] The server prioritizes the retrieved recipes based on the user's past selection history and preferences. For example, "grilled chicken and broccoli" and "tomato and chicken pasta" are listed as high-priority recipes.

[1014] Step 8:

[1015] The server generates a prioritized recipe list and transmits it to the user's terminal, where it is displayed in a format that the user can view.

[1016] Step 9:

[1017] The terminal will display a message to the user similar to:

[1018] We recommend the following recipes for dinner tonight:

[1019] 1. Grilled Chicken and Broccoli

[1020] 2. Tomato and chicken pasta

[1021] Which one would you like?

[1022] The user selects one of the suggested recipes.

[1023] Step 10:

[1024] The terminal receives the recipe selected by the user and transmits the information to the server.

[1025] Step 11:

[1026] The server generates detailed information about the selected recipe and sends it to the device, including cooking instructions, a list of ingredients, calorie information, and nutritional balance.

[1027] Step 12:

[1028] The device displays the detailed information received from the server to the user, allowing the user to check the specific cooking steps and ingredient information for the selected recipe.

[1029] Step 13:

[1030] After the user tries the recipe, they submit feedback from within the app. The device collects the user's feedback information and sends it to the server.

[1031] Step 14:

[1032] The server stores the received feedback in a database, analyzes the stored feedback information, and updates the recommendation algorithm to improve the accuracy of future recipe suggestions.

[1033] Example 1

[1034] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1035] Conventional recipe suggestion systems have problems in that they do not allow users to fully utilize the ingredients in their refrigerators, and they are unable to suggest recipes that fully reflect the user's preferences and past selection history. They also lack a means to improve the accuracy of recipe suggestions using user feedback. As a result, users spend a lot of time finding suitable recipes, and are unable to efficiently use the ingredients in their refrigerators.

[1036] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1037] In this invention, the server includes a means for a user to acquire an image of the contents of the refrigerator, an image recognition means for identifying the acquired image and identifying food items, a means for suggesting recipes based on information about the identified food items, a means for providing detailed information about the suggested recipes, a means for the user to provide feedback, and a means for accumulating the provided feedback and improving the recipe suggestion algorithm. This allows users to efficiently use ingredients in the refrigerator and receive recipe suggestions based on their individual preferences and past selection history. The accuracy of the recipe suggestions can also be improved based on the feedback.

[1038] A "user" is a person who uses the system to manage ingredients in the refrigerator and receive recipe suggestions.

[1039] "Means for acquiring images of the contents of the refrigerator" is a function that allows a user to take a photo of the food in the refrigerator using a smartphone or tablet and send the image data to the system.

[1040] "Image recognition means" refers to technology that analyzes acquired image data and identifies ingredients in the image. A deep learning-based algorithm is used.

[1041] "Food items" refers to ingredients or food stored in the refrigerator, such as tomatoes, chicken, and broccoli.

[1042] The "means for suggesting recipes" is a function that provides optimal cooking methods and cooking ideas based on information about the identified food items.

[1043] "Detailed information" refers to additional information about the suggested recipe, such as specific cooking instructions, a list of ingredients needed, calorie information, and nutritional balance.

[1044] "Feedback means" refers to a function that allows users to send ratings and comments on suggested recipes to the system.

[1045] "Means for accumulating feedback and improving the recipe suggestion algorithm" refers to a method for accumulating feedback data provided by users in a database and analyzing that data to improve the accuracy of the recipe suggestion algorithm.

[1046] "User's past choices and preferences" refers to the recipes and ingredients the user has chosen in the past, the types of dishes they like, and other information that is taken into consideration when suggesting recipes.

[1047] As an embodiment of the present invention, we provide a system in which a user scans the contents of their refrigerator using a smartphone or tablet, identifies food items using image recognition technology, and suggests optimal recipes based on that information. This system has the user, a terminal, and a server as its main elements, and these elements function in cooperation with each other.

[1048] User scans the contents of the refrigerator

[1049] The user launches the app installed on their smartphone. Using the app's camera function, the user takes a picture of the contents of the refrigerator. For example, imagine a scenario in which the user takes out "tomatoes," "chicken," and "broccoli" from the refrigerator and takes a picture of them. The smartphone acquires the captured image and sends it to the server via the HTTPS protocol.

[1050] The server performs image recognition

[1051] The server receives the image of the refrigerator interior sent by the user. The server is equipped with a deep learning-based image recognition module (e.g., TensorFlow, PyTorch) that recognizes food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. Information about the recognized food items is stored in a database (e.g., MySQL, MongoDB).

[1052] Server suggests recipes

[1053] The server searches a recipe database based on the recognized food item list. The recipe database contains a vast amount of recipe information, and searches for and suggests recipes related to specific food items. The server also takes into account the user's past selection history and preferences to filter and prioritize the search results. For example, recipes such as "grilled chicken and broccoli" and "pasta with tomato and chicken" are suggested.

[1054] The device notifies the user of recipe suggestions

[1055] The server sends the generated recipe list to the user's device. The smartphone app displays the suggested recipe list on the user's screen and notifies the user with an interactive message:

[1056] We recommend the following recipes for dinner tonight:

[1057] 1. Grilled Chicken and Broccoli

[1058] 2. Tomato and chicken pasta

[1059] Which one would you like?

[1060] User selects a recipe and receives details

[1061] The user selects one of the suggested recipes. For example, if the user selects "grilled chicken and broccoli," the smartphone sends the selection to the server. The server generates detailed information about the selected recipe and sends it to the device. The detailed information includes cooking instructions, a list of ingredients, calorie information, and nutritional balance. The device displays this information on the user's screen.

[1062] Users provide feedback

[1063] After trying the suggested recipes, users can provide feedback from within the app, such as "It was delicious" or "I wish it was a little saltier." The device then sends this feedback information to the server.

[1064] The server collects feedback and improves the algorithm

[1065] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[1066] Example prompts for generative AI models

[1067] Prompt statement:

[1068] "Design a system that analyzes a user's image of the inside of a refrigerator, identifies the food items contained within, and suggests optimal recipes based on that. Include recipe filtering that takes into account the user's past choices and preferences."

[1069] In this way, the system efficiently utilizes the contents of the refrigerator and significantly reduces the effort required for users to make dinner. For example, users can select a recipe based on the items in the refrigerator and then recreate it at home, saving time and effort.

[1070] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1071] Step 1:

[1072] A user scans the contents of the refrigerator with their smartphone

[1073] Input: App installed on smartphone, user's shooting operation

[1074] How it works: The user launches the app and uses the camera to take a picture of the food in the refrigerator. They can also take multiple images in succession.

[1075] Output: Image data of the inside of the refrigerator

[1076] Step 2:

[1077] The device sends the image to the server

[1078] Input: Captured image, smartphone network connection

[1079] What it does: The app sends the acquired image data to a server using the HTTPS protocol.

[1080] Output: Image data sent to the server

[1081] Step 3:

[1082] The server receives and stores the image

[1083] Input: Submitted image data

[1084] What it does: The server saves the image data in a temporary directory, specifically as a file, for later processing.

[1085] Output: Saved image file

[1086] Step 4:

[1087] The server performs image recognition

[1088] Input: Saved image file

[1089] How it works: The server's deep learning-based image recognition module analyzes the image and identifies the food item, using a Convolutional Neural Network (CNN) model (e.g., TensorFlow or PyTorch).

[1090] Output: A list of identified food items (e.g., "tomato," "chicken," "broccoli")

[1091] Step 5:

[1092] The server stores the food item list in a database.

[1093] Input: A list of identified food items

[1094] What it does: Stores the identified food item information in JSON format in a database (e.g. MySQL, MongoDB).

[1095] Output: Stored food item data

[1096] Step 6:

[1097] The server searches for and suggests recipes

[1098] Input: Stored food item data

[1099] How it works: Searches for recipes related to a specified food item in a database, filters the search results based on past history and preferences, and extracts relevant recipes using SQL queries.

[1100] Output: A list of suggested recipes (e.g. "Grilled Chicken and Broccoli" or "Pasta with Tomato and Chicken")

[1101] Step 7:

[1102] The device notifies the user of the recipe list

[1103] Input: Suggested recipe list

[1104] Operation: The smartphone app displays the recipe list received from the server in the form of a message to the user.

[1105] Output: Recipe selection message screen

[1106] Step 8:

[1107] The user selects a recipe

[1108] Input: User selection of recipe

[1109] How it works: The user selects one of the suggested recipes on the screen, and the selection is sent to the server.

[1110] Output: The ID of the selected recipe.

[1111] Step 9:

[1112] The server generates and sends the recipe details.

[1113] Input: ID of the selected recipe

[1114] What it does: Retrieves detailed information about the selected recipe from the recipe database (e.g., cooking instructions, ingredient list, calorie information, etc.) and sends the information in JSON format to the smartphone.

[1115] Output: Recipe details

[1116] Step 10:

[1117] The device displays detailed information to the user

[1118] Input: Recipe details received from the server

[1119] What it does: The app parses the details and displays them on the screen in a user-friendly format.

[1120] Output: Recipe details screen displayed

[1121] Step 11:

[1122] Users provide feedback

[1123] Input: User-initiated feedback input

[1124] How it works: The user enters feedback about a recipe and the app sends that data to the server.

[1125] Output: Feedback data sent

[1126] Step 12:

[1127] The server collects feedback and improves the algorithm

[1128] Input: Received feedback data

[1129] How it works: The server stores the feedback data in a database, analyzes it, and improves the recipe suggestion algorithm. It also retrains the machine learning model to reflect this in future recipe suggestions.

[1130] Output: An improved recipe suggestion algorithm

[1131] (Application example 1)

[1132] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1133] In recent years, food waste and shortening cooking times at home have become social issues. There is a demand for efficient use of refrigerator contents and reduction of waste, but current technology requires the time-consuming manual management of food in the refrigerator, making it difficult to find appropriate recipes based on that information. Furthermore, if necessary ingredients are in short supply, users have to go shopping, which is a significant burden on them. There is a need for a system that can solve these issues.

[1134] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1135] In this invention, the server includes means for a user to acquire an image of the contents of the refrigerator, image recognition means for identifying the acquired image and specifying food items, means for suggesting recipes based on information about the identified food items, means for providing detailed information about the suggested recipes, and means for automatically listing missing ingredients and ordering the ingredients from a delivery service, thereby enabling efficient use of the contents of the refrigerator and saving time and effort.

[1136] A "user" is an individual or group that uses this system to manage the contents of their refrigerator and receive recipe suggestions.

[1137] A "refrigerator" is a household or commercial appliance used to store food at the appropriate temperature.

[1138] "Images" are photos of food in the refrigerator taken by the user using a smartphone or tablet.

[1139] "Acquisition" means that a user takes a picture of the inside of the refrigerator using a smartphone or tablet and imports that image data into the system.

[1140] "Ingredient items" are specific foods and ingredients stored in the refrigerator.

[1141] "Image recognition" is a technology that analyzes captured images and identifies the food items contained within them.

[1142] "Recipe suggestion" refers to providing a cooking method suitable for the user based on information about the identified ingredient items.

[1143] "Detailed information" refers to information about the proposed recipe, including specific cooking steps, necessary ingredients, calorie information, and the like.

[1144] A "missing ingredient" is an ingredient item that is needed to create a suggested recipe but is not present in the refrigerator.

[1145] "Delivery service" is a service that allows users to order missing materials online and have these materials delivered promptly.

[1146] "Automatic Listing" is a function that automatically generates a list of ingredients that are in short supply based on recipe suggestions.

[1147] An "order" is a series of actions and procedures in which a user requests a delivery service to provide missing materials and arranges for delivery.

[1148] To implement this invention, a user uses a smartphone to scan the contents of their refrigerator, and the server uses image recognition technology to identify food items and suggests optimal recipes based on that information. Furthermore, the system automatically lists any missing ingredients and allows the user to order those ingredients via a delivery service.

[1149] Hardware and Software Configuration

[1150] Hardware:

[1151] On the user's phone:

[1152] Use the camera feature to scan the contents of your refrigerator.

[1153] software:

[1154] server:

[1155] Use Deep Learning-based frameworks such as TensorFlow / Keras for image recognition.

[1156] PIL (Python Imaging Library) is used for image processing.

[1157] The requests library is used to process HTTP requests.

[1158] Terminal app:

[1159] The contents of the refrigerator are scanned and the images are sent to a server.

[1160] The suggested recipe and the list of ingredients that are in short supply received from the server are displayed to the user.

[1161] Call the delivery service API to order the required ingredients.

[1162] System Operation

[1163] Fridge scanning and image recognition

[1164] The user launches the "Refrigerator-Linked Delivery Assistant" app on their smartphone and uses the camera to take a picture of the food in their refrigerator. The image data is sent from the device app to the server. The server uses a deep learning-based image recognition system to identify the food items in the image and generate a list of identified items. For example, "tomatoes," "chicken," and "onions" in the refrigerator are recognized.

[1165] Recipe suggestions and a list of missing ingredients

[1166] Based on the recognized food item list, the server searches a recipe database and suggests several recipes suitable for the user, taking into account the user's past selection history and preferences. The server also automatically lists any ingredients needed for the suggested recipes that are missing from the refrigerator. For example, a recipe for "Tomato Chicken Salad" may be suggested, but the server will point out that the required "lettuce" is missing.

[1167] Automating delivery orders

[1168] The list of missing ingredients is sent from the device app to the delivery service API, which automatically places an order, allowing users to get the missing ingredients immediately. Once the delivery order is successfully processed, the order status is displayed on the device.

[1169] Specific examples

[1170] For example, suppose a user scans the "tomatoes," "chicken," and "onions" in their refrigerator. Based on this, the server suggests a recipe for "Tomato Chicken Salad" and lists the missing ingredient, "lettuce," which is needed but not present. The server then automatically orders "lettuce" using a delivery service API, and the user can receive the ingredients via delivery.

[1171] Prompt Sentence Examples

[1172] text

[1173] Scan the image of the refrigerator

[1174] recipes, missing_items = scan_fridge('fridge_image.jpg')

[1175] Create a delivery order

[1176] if missing_items:

[1177] order_status = order_missing_items(missing_items)

[1178] print("Order Status:", order_status)

[1179] View suggested recipes

[1180] if recipes:

[1181] for idx, recipe in enumerate(recipes):

[1182] print(f"Recipe {idx + 1}:{recipe['name']}")

[1183] print(f"ingredients:{recipe['ingredients']}")

[1184] print(f"Steps:{recipe['steps']}")

[1185] print("\n")

[1186] In this way, by efficiently managing food items in the refrigerator and easily obtaining the necessary ingredients, users can significantly save time and effort in cooking.

[1187] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1188] Step 1:

[1189] The user launches the "Refrigerator-Linked Delivery Assistant" app on their smartphone and uses the camera to take a picture of the inside of the refrigerator. The image is then temporarily saved in the smartphone's local storage.

[1190] Input: Image of the inside of the refrigerator taken with a smartphone camera

[1191] Output: Images saved in the smartphone's local storage

[1192] Specific operation: The user launches the app, takes a picture of the inside of the refrigerator with their smartphone camera, and saves the image.

[1193] Step 2:

[1194] The device app sends the saved image to the server, which receives it as an HTTP request.

[1195] Input: A picture of a refrigerator stored on a smartphone

[1196] Output: Image data sent to the server

[1197] Specific operation: The terminal application obtains the image data and sends the image to the server via an HTTP request.

[1198] Step 3:

[1199] The server inputs the received image data into a deep learning-based image recognition system to identify food items, and the list of identified items is stored in a database on the server.

[1200] Input: Image data sent to the server

[1201] Output: A list of identified food items

[1202] What it does: The server uses TensorFlow / Keras to analyze the image and identify the food items in it, resulting in a list of items.

[1203] Step 4:

[1204] The server searches a recipe database based on the recognized food item list and suggests multiple recipes suitable for the user. The suggested recipes are generated as a list, taking into account the user's past selection history and preferences.

[1205] Input: List of identified food items

[1206] Output: A list of suggested recipes

[1207] What it does: The server runs a database query to find relevant recipes and prioritizes them based on the user's historical data.

[1208] Step 5:

[1209] The server creates a list of missing ingredients based on the proposed recipe and sends it to the device app, which then passes the list to the delivery service API.

[1210] Input: Suggested recipe list

[1211] Output: List of missing materials

[1212] What it does: The server matches the suggested recipe with the current food items, lists any missing ingredients, and sends them to the delivery service API.

[1213] Step 6:

[1214] The terminal app displays the list of missing ingredients received from the server to the user, automatically places a delivery service order, and notifies the user of the order status of the missing ingredients.

[1215] Input: List of materials in short supply received from the server

[1216] Output: The list displayed to the user and the order status

[1217] What it does: The device app displays a list of missing ingredients and updates the user on the status of their delivery order in real time.

[1218] Step 7:

[1219] The user selects a suggested recipe and receives detailed information about it on their device, including cooking instructions, ingredients needed, and calorie information.

[1220] Input: Suggested recipe list

[1221] Output: Detailed information about the selected recipe

[1222] Specific behavior: The user selects a suggested recipe within the app and receives detailed information from the server.

[1223] Step 8:

[1224] After cooking, users provide feedback through the device app, which is then sent to the server and used to suggest recipes for future meals.

[1225] Input: User feedback

[1226] Output: Feedback data stored on the server

[1227] How it works: The user enters feedback into the device app and sends it to the server, which stores it in a database and uses it to improve the accuracy of the algorithm.

[1228] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1229] This invention provides a system that allows a user to scan the contents of their refrigerator using a smartphone or tablet, and then combines image recognition technology with emotion recognition technology to identify food items and suggest optimal recipes based on the user's emotional state. Below, the program's processing is explained in natural language using concrete examples.

[1230] User scans the contents of the refrigerator

[1231] The user launches the "Dinner Omakase" app installed on their smartphone. They then use the app's camera function to take a picture of the contents of their refrigerator. The device then acquires the captured image data (image of the inside of the refrigerator) and sends this image to the server.

[1232] The server performs image recognition

[1233] The server receives the image of the refrigerator interior sent by the user. A deep learning-based image recognition module is installed on the server, and this module is used to recognize food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. The list of identified food items is stored in a database.

[1234] The server performs emotion recognition.

[1235] When a user is using a device with a camera, the device captures the user's facial expression data and sends the data to an emotion recognition engine in the server. The emotion recognition engine analyzes the facial expression data and identifies the user's emotional state (e.g., "satisfied," "fatigued," "stressed," etc.).

[1236] Server suggests recipes

[1237] The server searches a recipe database based on the identified food item list and emotion recognition results to retrieve relevant recipes. The recipe database contains a vast amount of recipe information, and searches for and suggests recipes that are suitable for the identified food items and the user's emotional state. For example, if the user is in a "fatigue" state, recipes that are easy to make and can be made in a short time are prioritized.

[1238] The device notifies the user of recipe suggestions

[1239] The server generates a list of suggested recipes and sends it to the user's device. The device's app displays the list of suggested recipes on the user's screen and sends the following interactive message:

[1240] We recommend the following recipes for dinner tonight:

[1241] 1. Grilled Chicken and Broccoli (Cooking time: 20 minutes)

[1242] 2. Tomato and Chicken Pasta (Cooking time: 15 minutes)

[1243] Which one would you like?

[1244] The user selects one of the suggested recipes.

[1245] User selects a recipe and receives details

[1246] The device receives the recipe selected by the user and sends that information to the server. The server generates detailed information about the selected recipe and sends it to the device. The detailed information includes cooking instructions, a list of ingredients, calorie information, and nutritional balance. The device displays this information on the user's screen.

[1247] Users provide feedback

[1248] After trying the suggested recipes, users can provide feedback from within the app, such as "It was delicious" or "I wish it was a little saltier." The device then sends this feedback information to the server.

[1249] The server collects feedback and improves the algorithm

[1250] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[1251] In this way, by combining emotion recognition technology, the system can quickly suggest optimal recipes based on the user's current emotional state, significantly reducing the effort and stress of cooking dinner. As a specific example, a user can select a recipe based on the items in the refrigerator and their own emotional state, and then actually recreate it at home, saving time and effort.

[1252] The processing flow will be explained below.

[1253] Step 1:

[1254] The user launches the "Dinner Omakase" app on their smartphone and uses the app's camera function to take a picture of the contents of the refrigerator. The device temporarily saves the image as a "refrigerator image."

[1255] Step 2:

[1256] The terminal sends the "refrigerator interior image" to the server, and the image data is transferred to the server via the network.

[1257] Step 3:

[1258] The server receives the "refrigerator image" sent by the user. The received image data is passed to the image recognition module in the server.

[1259] Step 4:

[1260] The server's image recognition module uses deep learning-based algorithms to analyze and identify food items in the image, such as "tomato," "chicken," and "broccoli."

[1261] Step 5:

[1262] The server generates a list of the identified food items and stores it in a database. The identified food item information is stored in the database.

[1263] Step 6:

[1264] If the user is using a device with a camera, the device captures the user's facial expression data and sends the data to an emotion recognition engine.

[1265] Step 7:

[1266] The server's emotion recognition engine analyzes the facial expression data and identifies the user's emotional state (e.g., "satisfied," "fatigued," "stressed," etc.) This emotional state information is stored in a database.

[1267] Step 8:

[1268] The server searches the recipe database based on the food item list and emotion recognition results to retrieve related recipes. Multiple recipes are found in the recipe database.

[1269] Step 9:

[1270] The server prioritizes the retrieved recipes based on the user's past selection history, preferences, and current emotional state. For example, if the user is "fatigued," recipes that are easy to prepare and can be made quickly will be prioritized.

[1271] Step 10:

[1272] The server generates a prioritized recipe list and transmits it to the user's terminal, where it is displayed in a format that the user can view.

[1273] Step 11:

[1274] The terminal will display a message to the user similar to:

[1275] We recommend the following recipes for dinner tonight:

[1276] 1. Grilled Chicken and Broccoli (Cooking time: 20 minutes)

[1277] 2. Tomato and Chicken Pasta (Cooking time: 15 minutes)

[1278] Which one would you like?

[1279] The user selects one of the suggested recipes.

[1280] Step 12:

[1281] The terminal receives the recipe selected by the user and transmits the information to the server.

[1282] Step 13:

[1283] The server generates detailed information about the selected recipe and sends it to the device, including cooking instructions, a list of ingredients, calorie information, and nutritional balance.

[1284] Step 14:

[1285] The device displays the detailed information received from the server to the user, allowing the user to check the specific cooking steps and ingredient information for the selected recipe.

[1286] Step 15:

[1287] After the user tries the recipe, they submit feedback from within the app. The device collects the user's feedback information and sends it to the server.

[1288] Step 16:

[1289] The server stores the received feedback in a database, analyzes the stored feedback information, and updates the recommendation algorithm to improve the accuracy of future recipe suggestions.

[1290] Example 2

[1291] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1292] Conventional food management and recipe recommendation systems have difficulty not only accurately identifying the food in a user's refrigerator, but also automating optimal recipe recommendations that take into account the user's emotional state. This has left users unable to quickly choose an appropriate dish that suits their current mood while utilizing the food in their refrigerator, often resulting in stress when preparing dinner. To solve this problem, a recipe recommendation system that simultaneously considers the food in the refrigerator and the user's emotional state is needed.

[1293] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1294] In this invention, the server includes means for allowing a user to acquire an image of the contents of the refrigerator, image recognition means for identifying the acquired image and specifying food items, emotion recognition means for acquiring data on the user's facial expression and analyzing the data to identify the user's emotional state, means for suggesting recipes based on information on the identified food items and the user's emotional state, and means for providing detailed information on the suggested recipes. This makes it possible to quickly suggest optimal recipes based on information on the food items in the refrigerator and the user's current emotional state.

[1295] "User" refers to an individual who uses the system to scan the contents of their refrigerator and receive recipe suggestions.

[1296] "Means of acquiring images" refers to the process in which a user takes a photo of the food in the refrigerator using the camera function of a smartphone or other device.

[1297] "Image recognition means" refers to technology for analyzing acquired image data and identifying food items in the refrigerator.

[1298] "Facial expression data" refers to image data captured by a camera of a user's face, and is used for emotion recognition.

[1299] "Emotion recognition means" refers to technology that analyzes facial expression data and identifies the user's emotional state (e.g., "satisfaction," "fatigue," "stress," etc.).

[1300] "Means for suggesting recipes" refers to a function that selects and suggests appropriate recipes based on information about the identified food items and the user's emotional state.

[1301] "Means for providing detailed information" refers to a function that provides users with detailed information such as cooking instructions for the suggested recipe, a list of required ingredients, calorie information, and nutritional balance.

[1302] The present invention is a system that allows a user to scan the contents of a refrigerator using a smartphone or tablet, and combines image recognition technology and emotion recognition technology to identify food items and suggest optimal recipes based on the user's emotional state. Next, a specific embodiment of this system will be described.

[1303] The system uses the following hardware and software:

[1304] Mobile devices such as smartphones and tablets

[1305] Camera on the device

[1306] Deep learning-based image recognition modules (e.g., TensorFlow and PyTorch)

[1307] Emotion recognition engine (e.g., Microsoft Azure Face API)

[1308] Recipe Database

[1309] Specific steps

[1310] User scans the contents of the refrigerator

[1311] The user launches the "Dinner Omakase" app installed on their smartphone and uses the camera to take a picture of the contents of the refrigerator. The device acquires the captured image data and sends it to the server in real time.

[1312] The server performs image recognition

[1313] The server receives the image of the refrigerator sent by the user and uses a deep learning-based image recognition module to recognize food items in the image, such as "tomato," "chicken," and "broccoli," and stores this information in a database.

[1314] The server performs emotion recognition.

[1315] When a user is using a device with a camera, the device captures the user's facial expression data and sends it to an emotion recognition engine in the server. The server analyzes the facial expression data and identifies the user's emotional state (e.g., "satisfied," "fatigued," "stressed," etc.).

[1316] Server suggests recipes

[1317] The server searches the recipe database based on the identified food item list and the emotion recognition results to retrieve relevant recipes. For example, if the user is in a "fatigue" state, it will prioritize recipes that are easy to make and can be made in a short time.

[1318] The device notifies the user of recipe suggestions

[1319] The server generates a list of suggested recipes and sends it to the user's device, where the app displays the list on the screen and provides specific lists and options.

[1320] User selects a recipe and receives more information

[1321] The user selects one of the suggested recipes. The selected recipe information is sent from the device to the server, and the server generates detailed cooking instructions, a list of ingredients, calorie information, nutritional balance, and other information and sends it to the device. The device then displays this information on the user's screen.

[1322] Users provide feedback

[1323] After trying the suggested recipes, users provide feedback from within the app, which the device then captures and sends to the server.

[1324] The server accumulates feedback and improves the algorithm

[1325] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[1326] Examples of prompt statements

[1327] For example, if a user uses a "Dinner Picker" app to scan the items "clams" and "garlic" in the refrigerator, and the app recognizes the user's facial expression as indicating a "fatigue" state, the prompt sentence based on this example might be:

[1328] Use the "Dinner Omakase" app to scan the contents of your refrigerator (clams, garlic) and suggest the best dinner recipe through image and emotion recognition. The user's emotional state is "fatigue."

[1329] In this way, by combining emotion recognition technology, the system can quickly suggest optimal recipes based on the user's current emotional state, significantly reducing the effort and stress of cooking dinner.

[1330] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1331] Processing flow

[1332] Step 1:

[1333] The user launches the app installed on their smartphone and takes a picture of the contents of the refrigerator. At this time, the user uses the camera function in the app to capture image data (images of the food in the refrigerator). The image data obtained as input is saved in the device's internal storage. Specifically, the user taps the camera icon on the app screen and takes a picture of the inside of the refrigerator with the smartphone's camera.

[1334] Step 2:

[1335] The device sends the acquired image data to the server. At this time, the device's app uploads the image data to the server via an Internet connection. The input is the image data acquired in step 1, and the output is the image data sent to the server. Specifically, the device uploads the data to the server using a background process.

[1336] Step 3:

[1337] The server receives image data sent by the user and identifies food items using an image recognition module. The image data received as input is analyzed using a deep learning-based image recognition algorithm (e.g., TensorFlow or PyTorch) to generate a list of food items. Specifically, the server calls the image recognition API and generates a list of items such as "tomato," "chicken," and "broccoli."

[1338] Step 4:

[1339] The server stores the identified food item list in a database, where the input is the food item list obtained in step 3 and the output is the information stored in the database. Specifically, the server opens a database connection and executes an SQL query to store the listed food items.

[1340] Step 5:

[1341] When a user is using a device with a camera, the device acquires the user's facial expression data and sends it to the emotion recognition engine. The input is the user's facial expression image data, and the output is the transmission of facial expression data from the device to the server. Specifically, the user shows an expression in front of the camera, and the device captures it and uploads it to the server.

[1342] Step 6:

[1343] The server analyzes the received facial expression data using an emotion recognition engine to identify the user's emotional state. The input is facial expression data, and the output is the identified emotional state (e.g., "satisfied," "fatigued," "stressed," etc.). Specifically, the server calls an emotion recognition API (e.g., Microsoft Azure Face API) to analyze the emotional state.

[1344] Step 7:

[1345] The server searches the recipe database based on the identified food item list and emotion recognition results to suggest optimal recipes. The input is the food item list and the emotional state, and the output is a list of related recipes. Specifically, the server generates a query and executes an SQL query to retrieve suitable recipes from the recipe database.

[1346] Step 8:

[1347] The server generates a list of suggested recipes and sends it to the user's device. The input is the recipe list, and the output is sending the recipe list to the device. Specifically, the server sends the generated recipe list to the device via push notification.

[1348] Step 9:

[1349] The device displays the suggested recipe list on the user's screen. The input is the recipe list sent from the server, and the output is the list displayed on the user's screen. In concrete terms, the app receives the recipe list and displays it using a UI component.

[1350] Step 10:

[1351] The user selects one of the suggested recipes. The input is the displayed recipe list, and the output is the selected recipe information. Specifically, the user selects a recipe by tapping on the screen.

[1352] Step 11:

[1353] The terminal sends the selected recipe information to the server. The input is the recipe information selected by the user, and the output is the sending of a recipe list to the server. In concrete terms, the terminal uploads the selected content to the server.

[1354] Step 12:

[1355] The server generates detailed information for the selected recipe and sends it to the terminal. The input is the selected recipe information, and the output is the detailed recipe information. Specifically, the server generates detailed cooking instructions and ingredient lists and sends them to the terminal.

[1356] Step 13:

[1357] The device displays detailed recipe information on the user's screen. The input is the detailed recipe information sent from the server, and the output is the detailed information displayed on the user's screen. Specifically, the app receives the detailed information and displays it as a description or list.

[1358] Step 14:

[1359] The user enters feedback within the app. The input is the user's feedback comment, and the output is saving the feedback information to the device. Specifically, the user writes a comment in the feedback form and taps the submit button.

[1360] Step 15:

[1361] The terminal sends feedback information to the server. The input is the feedback information provided by the user, and the output is the transmission of the feedback information to the server. In concrete terms, the terminal uploads the transmitted feedback to the server.

[1362] Step 16:

[1363] The server accumulates the feedback sent by the user in a database and reflects it in future recipe suggestions. The input is the feedback data, and the output is an updated recipe suggestion algorithm. Specifically, the server stores the feedback data in storage and uses it to improve the machine learning algorithm.

[1364] (Application example 2)

[1365] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1366] Conventional recipe suggestion systems could suggest recipes based on the contents of a user's refrigerator, but they could not take the user's emotional state into account. This made it difficult for users to find products and recipes that matched their emotions at any given time. The present invention aims to solve these problems by providing a system that detects a user's emotional state and suggests optimal products and recipes based on that emotional state.

[1367] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1368] In this invention, the server includes means for a user to acquire an image of the contents of the refrigerator, image recognition means for identifying food items by recognizing the acquired image, means for suggesting recipes based on information on the identified food items, means for providing detailed information on the suggested recipes, emotion recognition means for scanning the user's facial expression to identify their emotional state, means for recommending products based on the user's emotional state, and means for providing detailed information on the recommended products, thereby enabling the user to receive recommendations that are optimal for their emotional state at the time.

[1369] "User" refers to an individual who uses the system.

[1370] "Means for acquiring an image of the contents of the refrigerator" refers to a function that allows a user to take a picture of the contents of the refrigerator using the camera on a smartphone or tablet and acquire the image data.

[1371] "Image recognition means" refers to technology for analyzing captured image data and identifying food items present in the image.

[1372] "Food items" refer to ingredients or foods present in an image, including specific items such as tomatoes or chicken.

[1373] The "means for suggesting recipes" refers to a function that provides the user with appropriate cooking methods and menus based on information about food items identified by the image recognition means.

[1374] "Means for providing detailed information" refers to a function that presents the user with specific cooking steps, a list of ingredients, calorie information, etc. for a recipe.

[1375] "Emotion recognition means for scanning facial expressions to identify emotional states" refers to technology for analyzing a user's facial expression data and identifying their emotional state at that time (e.g., joy, fatigue, etc.).

[1376] "Means for recommending products" refers to a function that suggests appropriate products to users based on their emotional state and past behavioral history.

[1377] "Means for providing detailed information about recommended products" refers to a function that displays detailed information about recommended products, such as descriptions, prices, and reviews, to users.

[1378] The present invention relates to a system that allows users to scan the contents of their refrigerator using a smartphone or tablet, combines image recognition technology with emotion recognition technology to identify food items, and suggests optimal recipes and products based on the user's emotional state.

[1379] System configuration

[1380] The system consists of the following main components:

[1381] 1. User device: A device equipped with a camera and display, such as a smartphone or tablet.

[1382] 2. Server: A cloud service that performs image recognition, emotion recognition, and recipe and product suggestions.

[1383] 3. Database: Stores food items, emotional states, recipes, product information, user behavior history and feedback.

[1384] Program execution procedure

[1385] 1. User scans the contents of the refrigerator

[1386] The user launches the dedicated application and takes a picture of the contents of the refrigerator using the camera function of their smartphone or tablet. The device acquires the image data and sends it to the server.

[1387] 2. Image recognition by the server

[1388] The server receives the image of the refrigerator sent by the user. The server is equipped with a deep learning-based image recognition module, which recognizes food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. The list of identified food items is stored in a database.

[1389] 3. Emotion recognition

[1390] When a user uses a device with a camera, the device captures the user's facial expression data and sends it to the server. The emotion recognition engine in the server analyzes the facial expression data and identifies the user's emotional state. For example, emotional states such as "joy," "fatigue," and "stress" are analyzed.

[1391] 4. Recipe and product suggestions

[1392] The server searches the recipe and product database based on the identified food item list and emotion recognition results to retrieve related recipes and products. Related recipes and products are suggested based on the user's emotional state. For example, if the user is "fatigued," recipes that are easy to make in a short time and products that help them relax are suggested first.

[1393] 5. Notifications on user devices

[1394] The server generates suggested recipes and product lists and sends them to the user's device. The device app displays the suggested recipes and product lists on the user's screen and sends interactive messages. For example, the following message might be displayed:

[1395] Here are some products we recommend for you right now:

[1396] 1. Grilled Chicken and Broccoli (Cooking time: 20 minutes)

[1397] 2. Tomato and Chicken Pasta (Cooking time: 15 minutes)

[1398] 3. Stress Relief Ball

[1399] 4. Herbal tea

[1400] Which one would you like?

[1401] Hardware and software used

[1402] Hardware:

[1403] Smartphone or tablet camera: Used to capture images of the user's facial expressions and the inside of the refrigerator.

[1404] software:

[1405] Python: The implementation language for the entire program.

[1406] opencv-python: Used to process facial expression and fridge image data.

[1407] Pillow: A library for image processing.

[1408] numpy: Used to process data.

[1409] TensorFlow: Emotion recognition using pre-trained models.

[1410] Prompt Sentence Examples

[1411] Write code to analyze a user's emotional state using facial images and recommend products that best match their emotions. Use the camera function to capture facial images and use TensorFlow to recognize emotions with a pre-trained model. Write a set of Python code to display a list of products based on the recognized emotion.

[1412] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1413] Step 1:

[1414] The user launches an application on their smartphone or tablet and uses the camera function to take a picture of the contents of the refrigerator. The device acquires the captured image data (image of the inside of the refrigerator) and sends this image to the server. The input is the image data captured by the user, and the output is the transfer of the image data to the server.

[1415] Step 2:

[1416] The server receives an image of the inside of the refrigerator sent by the user. It uses a deep learning-based image recognition module to recognize food items in the image. The input is image data of the inside of the refrigerator, and the output is a list of identified food items. Specifically, the image data is preprocessed and input into a model to identify food items.

[1417] Step 3:

[1418] The identified food item list is saved in a database. The input is the food item list output from image recognition, and the output is the process of saving it to the database. Specifically, the type and quantity of food items are written to the database.

[1419] Step 4:

[1420] The user takes a photo of their own facial expression using the camera function of their smartphone or tablet. The device acquires the facial expression data and sends it to the server. The input is the facial expression image taken by the user, and the output is the transfer of the facial expression data to the server.

[1421] Step 5:

[1422] The server receives the facial expression images and analyzes the facial expression data using an emotion recognition engine. The input is the facial expression image data, and the output is the user's emotional state. Specifically, the facial expression data is preprocessed and input into an emotion recognition model to identify the emotional state.

[1423] Step 6:

[1424] The server searches the recipe and product databases based on the identified food item list and emotional state to retrieve related recipes and products. The input is the food item list and emotional state, and the output is a list of recommended recipes and products. Specifically, it extracts information that matches the conditions from the recipe database and product database.

[1425] Step 7:

[1426] The server generates a list of recommended recipes and products and sends it to the user's device. The input is the recommendation list, and the output is data transmission to the user's device. Specifically, the server formats the selected information and delivers it to the user's device.

[1427] Step 8:

[1428] The user device displays the suggested recipes and product list on the screen and presents interactive messages to the user. For example, it presents multiple options along with the message "Here are the products we recommend for you right now:" The input is the recommendation list data from the server, and the output is the display to the user. Specifically, the information is presented visually using a GUI.

[1429] Step 9:

[1430] The user checks the detailed information of the suggested recipes and products and selects a specific recipe or product. The input is the user's selection, and the output is the transmission of the selected data to the server. Specifically, the system detects the user's action of pressing the selection button and sends the selected information to the server.

[1431] Step 10:

[1432] The server receives the user's selection information, generates detailed information about the selected recipe or product, and sends it to the user's device. The input is the user's selection information, and the output is a response with detailed information. Specifically, the server retrieves detailed information from the database, formats it, and provides it to the user.

[1433] Step 11:

[1434] The user terminal displays detailed information on the screen and provides cooking instructions and product details. The input is the detailed information data sent from the server, and the output is the display to the user. Specifically, the detailed information is presented visually using a GUI.

[1435] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1436] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1437] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1438] [Fourth embodiment]

[1439] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1440] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1441] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1442] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1443] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1444] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1445] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1446] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1447] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1448] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1449] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1450] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1451] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1452] As an embodiment of this invention, we provide a system that allows a user to scan the contents of their refrigerator using a smartphone or tablet, identifies food items using image recognition technology, and suggests optimal recipes based on that information. Next, we will explain the program's processing in natural language using concrete examples.

[1453] User scans the contents of the refrigerator

[1454] The user launches the "Dinner Omakase" app installed on their smartphone. Using the app's camera function, the user takes a picture of the contents of the refrigerator. The smartphone acquires the image and sends it to the server.

[1455] The server performs image recognition

[1456] The server receives the image of the refrigerator interior sent by the user. A deep learning-based image recognition module is installed on the server, and this module is used to recognize food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. The list of identified food items is stored in a database.

[1457] Server suggests recipes

[1458] The server searches a recipe database based on the identified food item list. The recipe database contains a vast amount of recipe information, and searches for and suggests recipes related to the specific food items. The server then filters and prioritizes the search results, taking into account the user's past selection history and preferences. For example, recipes such as "grilled chicken and broccoli" and "pasta with tomato and chicken" are suggested.

[1459] The device notifies the user of recipe suggestions

[1460] The server generates a list of suggested recipes and sends it to the user's device. The device's app displays the list of suggested recipes on the user's screen and sends the following interactive message:

[1461] We recommend the following recipes for dinner tonight:

[1462] 1. Grilled Chicken and Broccoli

[1463] 2. Tomato and chicken pasta

[1464] Which one would you like?

[1465] User selects a recipe and receives details

[1466] The user selects one of the suggested recipes. For example, if the user selects "grilled chicken and broccoli," the device sends the selection to the server. The server generates detailed information about the selected recipe and sends it to the device. The detailed information includes cooking instructions, a list of ingredients, calorie information, and nutritional balance. The device displays this information on the user's screen.

[1467] Users provide feedback

[1468] After trying the suggested recipes, users can provide feedback from within the app, such as "It was delicious" or "I wish it was a little saltier." The device then sends this feedback information to the server.

[1469] The server collects feedback and improves the algorithm

[1470] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[1471] As described above, this system efficiently utilizes the contents of the refrigerator and significantly reduces the effort required for users to make dinner. As a specific example, the user can select a recipe based on the items in the refrigerator and then recreate it at home, saving time and effort.

[1472] The processing flow will be explained below.

[1473] Step 1:

[1474] The user launches the "Dinner Omakase" app on their smartphone and uses the app's camera function to take a picture of the contents of the refrigerator. The device temporarily saves the image as a "refrigerator image."

[1475] Step 2:

[1476] The terminal sends the "refrigerator interior image" to the server, and the image data is transferred to the server via the network.

[1477] Step 3:

[1478] The server receives the "refrigerator image" sent by the user. The received image data is passed to the image recognition module in the server.

[1479] Step 4:

[1480] The server's image recognition module uses deep learning-based algorithms to analyze and identify food items in the image, such as "tomato," "chicken," and "broccoli."

[1481] Step 5:

[1482] The server generates a list of the identified food items and stores it in a database. The identified food item information is stored in the database.

[1483] Step 6:

[1484] The server searches the recipe database based on the food item list to retrieve related recipes. Multiple recipes are found in the recipe database.

[1485] Step 7:

[1486] The server prioritizes the retrieved recipes based on the user's past selection history and preferences. For example, "grilled chicken and broccoli" and "tomato and chicken pasta" are listed as high-priority recipes.

[1487] Step 8:

[1488] The server generates a prioritized recipe list and transmits it to the user's terminal, where it is displayed in a format that the user can view.

[1489] Step 9:

[1490] The terminal will display a message to the user similar to:

[1491] We recommend the following recipes for dinner tonight:

[1492] 1. Grilled Chicken and Broccoli

[1493] 2. Tomato and chicken pasta

[1494] Which one would you like?

[1495] The user selects one of the suggested recipes.

[1496] Step 10:

[1497] The terminal receives the recipe selected by the user and transmits the information to the server.

[1498] Step 11:

[1499] The server generates detailed information about the selected recipe and sends it to the device, including cooking instructions, a list of ingredients, calorie information, and nutritional balance.

[1500] Step 12:

[1501] The device displays the detailed information received from the server to the user, allowing the user to check the specific cooking steps and ingredient information for the selected recipe.

[1502] Step 13:

[1503] After the user tries the recipe, they submit feedback from within the app. The device collects the user's feedback information and sends it to the server.

[1504] Step 14:

[1505] The server stores the received feedback in a database, analyzes the stored feedback information, and updates the recommendation algorithm to improve the accuracy of future recipe suggestions.

[1506] Example 1

[1507] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1508] Conventional recipe suggestion systems have problems in that they do not allow users to fully utilize the ingredients in their refrigerators, and they are unable to suggest recipes that fully reflect the user's preferences and past selection history. They also lack a means to improve the accuracy of recipe suggestions using user feedback. As a result, users spend a lot of time finding suitable recipes, and are unable to efficiently use the ingredients in their refrigerators.

[1509] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1510] In this invention, the server includes a means for a user to acquire an image of the contents of the refrigerator, an image recognition means for identifying the acquired image and identifying food items, a means for suggesting recipes based on information about the identified food items, a means for providing detailed information about the suggested recipes, a means for the user to provide feedback, and a means for accumulating the provided feedback and improving the recipe suggestion algorithm. This allows users to efficiently use ingredients in the refrigerator and receive recipe suggestions based on their individual preferences and past selection history. The accuracy of the recipe suggestions can also be improved based on the feedback.

[1511] A "user" is a person who uses the system to manage ingredients in the refrigerator and receive recipe suggestions.

[1512] "Means for acquiring images of the contents of the refrigerator" is a function that allows a user to take a photo of the food in the refrigerator using a smartphone or tablet and send the image data to the system.

[1513] "Image recognition means" refers to technology that analyzes acquired image data and identifies ingredients in the image. A deep learning-based algorithm is used.

[1514] "Food items" refers to ingredients or food stored in the refrigerator, such as tomatoes, chicken, and broccoli.

[1515] The "means for suggesting recipes" is a function that provides optimal cooking methods and cooking ideas based on information about the identified food items.

[1516] "Detailed information" refers to additional information about the suggested recipe, such as specific cooking instructions, a list of ingredients needed, calorie information, and nutritional balance.

[1517] "Feedback means" refers to a function that allows users to send ratings and comments on suggested recipes to the system.

[1518] "Means for accumulating feedback and improving the recipe suggestion algorithm" refers to a method for accumulating feedback data provided by users in a database and analyzing that data to improve the accuracy of the recipe suggestion algorithm.

[1519] "User's past choices and preferences" refers to the recipes and ingredients the user has chosen in the past, the types of dishes they like, and other information that is taken into consideration when suggesting recipes.

[1520] As an embodiment of the present invention, we provide a system in which a user scans the contents of their refrigerator using a smartphone or tablet, identifies food items using image recognition technology, and suggests optimal recipes based on that information. This system has the user, a terminal, and a server as its main elements, and these elements function in cooperation with each other.

[1521] User scans the contents of the refrigerator

[1522] The user launches the app installed on their smartphone. Using the app's camera function, the user takes a picture of the contents of the refrigerator. For example, imagine a scenario in which the user takes out "tomatoes," "chicken," and "broccoli" from the refrigerator and takes a picture of them. The smartphone acquires the captured image and sends it to the server via the HTTPS protocol.

[1523] The server performs image recognition

[1524] The server receives the image of the refrigerator interior sent by the user. The server is equipped with a deep learning-based image recognition module (e.g., TensorFlow, PyTorch) that recognizes food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. Information about the recognized food items is stored in a database (e.g., MySQL, MongoDB).

[1525] Server suggests recipes

[1526] The server searches a recipe database based on the recognized food item list. The recipe database contains a vast amount of recipe information, and searches for and suggests recipes related to specific food items. The server also takes into account the user's past selection history and preferences to filter and prioritize the search results. For example, recipes such as "grilled chicken and broccoli" and "pasta with tomato and chicken" are suggested.

[1527] The device notifies the user of recipe suggestions

[1528] The server sends the generated recipe list to the user's device. The smartphone app displays the suggested recipe list on the user's screen and notifies the user with an interactive message:

[1529] We recommend the following recipes for dinner tonight:

[1530] 1. Grilled Chicken and Broccoli

[1531] 2. Tomato and chicken pasta

[1532] Which one would you like?

[1533] User selects a recipe and receives details

[1534] The user selects one of the suggested recipes. For example, if the user selects "grilled chicken and broccoli," the smartphone sends the selection to the server. The server generates detailed information about the selected recipe and sends it to the device. The detailed information includes cooking instructions, a list of ingredients, calorie information, and nutritional balance. The device displays this information on the user's screen.

[1535] Users provide feedback

[1536] After trying the suggested recipes, users can provide feedback from within the app, such as "It was delicious" or "I wish it was a little saltier." The device then sends this feedback information to the server.

[1537] The server collects feedback and improves the algorithm

[1538] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[1539] Example prompts for generative AI models

[1540] Prompt statement:

[1541] "Design a system that analyzes a user's image of the inside of a refrigerator, identifies the food items contained within, and suggests optimal recipes based on that. Include recipe filtering that takes into account the user's past choices and preferences."

[1542] In this way, the system efficiently utilizes the contents of the refrigerator and significantly reduces the effort required for users to make dinner. For example, users can select a recipe based on the items in the refrigerator and then recreate it at home, saving time and effort.

[1543] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1544] Step 1:

[1545] A user scans the contents of the refrigerator with their smartphone

[1546] Input: App installed on smartphone, user's shooting operation

[1547] How it works: The user launches the app and uses the camera to take a picture of the food in the refrigerator. They can also take multiple images in succession.

[1548] Output: Image data of the inside of the refrigerator

[1549] Step 2:

[1550] The device sends the image to the server

[1551] Input: Captured image, smartphone network connection

[1552] What it does: The app sends the acquired image data to a server using the HTTPS protocol.

[1553] Output: Image data sent to the server

[1554] Step 3:

[1555] The server receives and stores the image

[1556] Input: Submitted image data

[1557] What it does: The server saves the image data in a temporary directory, specifically as a file, for later processing.

[1558] Output: Saved image file

[1559] Step 4:

[1560] The server performs image recognition

[1561] Input: Saved image file

[1562] How it works: The server's deep learning-based image recognition module analyzes the image and identifies the food item, using a Convolutional Neural Network (CNN) model (e.g., TensorFlow or PyTorch).

[1563] Output: A list of identified food items (e.g., "tomato," "chicken," "broccoli")

[1564] Step 5:

[1565] The server stores the food item list in a database.

[1566] Input: A list of identified food items

[1567] What it does: Stores the identified food item information in JSON format in a database (e.g. MySQL, MongoDB).

[1568] Output: Stored food item data

[1569] Step 6:

[1570] The server searches for and suggests recipes

[1571] Input: Stored food item data

[1572] How it works: Searches for recipes related to a specified food item in a database, filters the search results based on past history and preferences, and extracts relevant recipes using SQL queries.

[1573] Output: A list of suggested recipes (e.g. "Grilled Chicken and Broccoli" or "Pasta with Tomato and Chicken")

[1574] Step 7:

[1575] The device notifies the user of the recipe list

[1576] Input: Suggested recipe list

[1577] Operation: The smartphone app displays the recipe list received from the server in the form of a message to the user.

[1578] Output: Recipe selection message screen

[1579] Step 8:

[1580] The user selects a recipe

[1581] Input: User selection of recipe

[1582] How it works: The user selects one of the suggested recipes on the screen, and the selection is sent to the server.

[1583] Output: The ID of the selected recipe.

[1584] Step 9:

[1585] The server generates and sends the recipe details.

[1586] Input: ID of the selected recipe

[1587] What it does: Retrieves detailed information about the selected recipe from the recipe database (e.g., cooking instructions, ingredient list, calorie information, etc.) and sends the information in JSON format to the smartphone.

[1588] Output: Recipe details

[1589] Step 10:

[1590] The device displays detailed information to the user

[1591] Input: Recipe details received from the server

[1592] What it does: The app parses the details and displays them on the screen in a user-friendly format.

[1593] Output: Recipe details screen displayed

[1594] Step 11:

[1595] Users provide feedback

[1596] Input: User-initiated feedback input

[1597] How it works: The user enters feedback about a recipe and the app sends that data to the server.

[1598] Output: Feedback data sent

[1599] Step 12:

[1600] The server collects feedback and improves the algorithm

[1601] Input: Received feedback data

[1602] How it works: The server stores the feedback data in a database, analyzes it, and improves the recipe suggestion algorithm. It also retrains the machine learning model to reflect this in future recipe suggestions.

[1603] Output: An improved recipe suggestion algorithm

[1604] (Application example 1)

[1605] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1606] In recent years, food waste and shortening cooking times at home have become social issues. There is a demand for efficient use of refrigerator contents and reduction of waste, but current technology requires the time-consuming manual management of food in the refrigerator, making it difficult to find appropriate recipes based on that information. Furthermore, if necessary ingredients are in short supply, users have to go shopping, which is a significant burden on them. There is a need for a system that can solve these issues.

[1607] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1608] In this invention, the server includes means for a user to acquire an image of the contents of the refrigerator, image recognition means for identifying the acquired image and specifying food items, means for suggesting recipes based on information about the identified food items, means for providing detailed information about the suggested recipes, and means for automatically listing missing ingredients and ordering the ingredients from a delivery service, thereby enabling efficient use of the contents of the refrigerator and saving time and effort.

[1609] A "user" is an individual or group that uses this system to manage the contents of their refrigerator and receive recipe suggestions.

[1610] A "refrigerator" is a household or commercial appliance used to store food at the appropriate temperature.

[1611] "Images" are photos of food in the refrigerator taken by the user using a smartphone or tablet.

[1612] "Acquisition" means that a user takes a picture of the inside of the refrigerator using a smartphone or tablet and imports that image data into the system.

[1613] "Ingredient items" are specific foods and ingredients stored in the refrigerator.

[1614] "Image recognition" is a technology that analyzes captured images and identifies the food items contained within them.

[1615] "Recipe suggestion" refers to providing a cooking method suitable for the user based on information about the identified ingredient items.

[1616] "Detailed information" refers to information about the proposed recipe, including specific cooking steps, necessary ingredients, calorie information, and the like.

[1617] A "missing ingredient" is an ingredient item that is needed to create a suggested recipe but is not present in the refrigerator.

[1618] "Delivery service" is a service that allows users to order missing materials online and have these materials delivered promptly.

[1619] "Automatic Listing" is a function that automatically generates a list of ingredients that are in short supply based on recipe suggestions.

[1620] An "order" is a series of actions and procedures in which a user requests a delivery service to provide missing materials and arranges for delivery.

[1621] To implement this invention, a user uses a smartphone to scan the contents of their refrigerator, and the server uses image recognition technology to identify food items and suggests optimal recipes based on that information. Furthermore, the system automatically lists any missing ingredients and allows the user to order those ingredients via a delivery service.

[1622] Hardware and Software Configuration

[1623] Hardware:

[1624] On the user's phone:

[1625] Use the camera feature to scan the contents of your refrigerator.

[1626] software:

[1627] server:

[1628] Use Deep Learning-based frameworks such as TensorFlow / Keras for image recognition.

[1629] PIL (Python Imaging Library) is used for image processing.

[1630] The requests library is used to process HTTP requests.

[1631] Terminal app:

[1632] The contents of the refrigerator are scanned and the images are sent to a server.

[1633] The suggested recipe and the list of ingredients that are in short supply received from the server are displayed to the user.

[1634] Call the delivery service API to order the required ingredients.

[1635] System Operation

[1636] Fridge scanning and image recognition

[1637] The user launches the "Refrigerator-Linked Delivery Assistant" app on their smartphone and uses the camera to take a picture of the food in their refrigerator. The image data is sent from the device app to the server. The server uses a deep learning-based image recognition system to identify the food items in the image and generate a list of identified items. For example, "tomatoes," "chicken," and "onions" in the refrigerator are recognized.

[1638] Recipe suggestions and a list of missing ingredients

[1639] Based on the recognized food item list, the server searches a recipe database and suggests several recipes suitable for the user, taking into account the user's past selection history and preferences. The server also automatically lists any ingredients needed for the suggested recipes that are missing from the refrigerator. For example, a recipe for "Tomato Chicken Salad" may be suggested, but the server will point out that the required "lettuce" is missing.

[1640] Automating delivery orders

[1641] The list of missing ingredients is sent from the device app to the delivery service API, which automatically places an order, allowing users to get the missing ingredients immediately. Once the delivery order is successfully processed, the order status is displayed on the device.

[1642] Specific examples

[1643] For example, suppose a user scans the "tomatoes," "chicken," and "onions" in their refrigerator. Based on this, the server suggests a recipe for "Tomato Chicken Salad" and lists the missing ingredient, "lettuce," which is needed but not present. The server then automatically orders "lettuce" using a delivery service API, and the user can receive the ingredients via delivery.

[1644] Prompt Sentence Examples

[1645] text

[1646] Scan the image of the refrigerator

[1647] recipes, missing_items = scan_fridge('fridge_image.jpg')

[1648] Create a delivery order

[1649] if missing_items:

[1650] order_status = order_missing_items(missing_items)

[1651] print("Order Status:", order_status)

[1652] View suggested recipes

[1653] if recipes:

[1654] for idx, recipe in enumerate(recipes):

[1655] print(f"Recipe {idx + 1}:{recipe['name']}")

[1656] print(f"ingredients:{recipe['ingredients']}")

[1657] print(f"Steps:{recipe['steps']}")

[1658] print("\n")

[1659] In this way, by efficiently managing food items in the refrigerator and easily obtaining the necessary ingredients, users can significantly save time and effort in cooking.

[1660] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1661] Step 1:

[1662] The user launches the "Refrigerator-Linked Delivery Assistant" app on their smartphone and uses the camera to take a picture of the inside of the refrigerator. The image is then temporarily saved in the smartphone's local storage.

[1663] Input: Image of the inside of the refrigerator taken with a smartphone camera

[1664] Output: Images saved in the smartphone's local storage

[1665] Specific operation: The user launches the app, takes a picture of the inside of the refrigerator with their smartphone camera, and saves the image.

[1666] Step 2:

[1667] The device app sends the saved image to the server, which receives it as an HTTP request.

[1668] Input: A picture of a refrigerator stored on a smartphone

[1669] Output: Image data sent to the server

[1670] Specific operation: The terminal application obtains the image data and sends the image to the server via an HTTP request.

[1671] Step 3:

[1672] The server inputs the received image data into a deep learning-based image recognition system to identify food items, and the list of identified items is stored in a database on the server.

[1673] Input: Image data sent to the server

[1674] Output: A list of identified food items

[1675] What it does: The server uses TensorFlow / Keras to analyze the image and identify the food items in it, resulting in a list of items.

[1676] Step 4:

[1677] The server searches a recipe database based on the recognized food item list and suggests multiple recipes suitable for the user. The suggested recipes are generated as a list, taking into account the user's past selection history and preferences.

[1678] Input: List of identified food items

[1679] Output: A list of suggested recipes

[1680] What it does: The server runs a database query to find relevant recipes and prioritizes them based on the user's historical data.

[1681] Step 5:

[1682] The server creates a list of missing ingredients based on the proposed recipe and sends it to the device app, which then passes the list to the delivery service API.

[1683] Input: Suggested recipe list

[1684] Output: List of missing materials

[1685] What it does: The server matches the suggested recipe with the current food items, lists any missing ingredients, and sends them to the delivery service API.

[1686] Step 6:

[1687] The terminal app displays the list of missing ingredients received from the server to the user, automatically places a delivery service order, and notifies the user of the order status of the missing ingredients.

[1688] Input: List of materials in short supply received from the server

[1689] Output: The list displayed to the user and the order status

[1690] What it does: The device app displays a list of missing ingredients and updates the user on the status of their delivery order in real time.

[1691] Step 7:

[1692] The user selects a suggested recipe and receives detailed information about it on their device, including cooking instructions, ingredients needed, and calorie information.

[1693] Input: Suggested recipe list

[1694] Output: Detailed information about the selected recipe

[1695] Specific behavior: The user selects a suggested recipe within the app and receives detailed information from the server.

[1696] Step 8:

[1697] After cooking, users provide feedback through the device app, which is then sent to the server and used to suggest recipes for future meals.

[1698] Input: User feedback

[1699] Output: Feedback data stored on the server

[1700] How it works: The user enters feedback into the device app and sends it to the server, which stores it in a database and uses it to improve the accuracy of the algorithm.

[1701] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1702] This invention provides a system that allows a user to scan the contents of their refrigerator using a smartphone or tablet, and then combines image recognition technology with emotion recognition technology to identify food items and suggest optimal recipes based on the user's emotional state. Below, the program's processing is explained in natural language using concrete examples.

[1703] User scans the contents of the refrigerator

[1704] The user launches the "Dinner Omakase" app installed on their smartphone. They then use the app's camera function to take a picture of the contents of their refrigerator. The device then acquires the captured image data (image of the inside of the refrigerator) and sends this image to the server.

[1705] The server performs image recognition

[1706] The server receives the image of the refrigerator interior sent by the user. A deep learning-based image recognition module is installed on the server, and this module is used to recognize food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. The list of identified food items is stored in a database.

[1707] The server performs emotion recognition.

[1708] When a user is using a device with a camera, the device captures the user's facial expression data and sends the data to an emotion recognition engine in the server. The emotion recognition engine analyzes the facial expression data and identifies the user's emotional state (e.g., "satisfied," "fatigued," "stressed," etc.).

[1709] Server suggests recipes

[1710] The server searches a recipe database based on the identified food item list and emotion recognition results to retrieve relevant recipes. The recipe database contains a vast amount of recipe information, and searches for and suggests recipes that are suitable for the identified food items and the user's emotional state. For example, if the user is in a "fatigue" state, recipes that are easy to make and can be made in a short time are prioritized.

[1711] The device notifies the user of recipe suggestions

[1712] The server generates a list of suggested recipes and sends it to the user's device. The device's app displays the list of suggested recipes on the user's screen and sends the following interactive message:

[1713] We recommend the following recipes for dinner tonight:

[1714] 1. Grilled Chicken and Broccoli (Cooking time: 20 minutes)

[1715] 2. Tomato and Chicken Pasta (Cooking time: 15 minutes)

[1716] Which one would you like?

[1717] The user selects one of the suggested recipes.

[1718] User selects a recipe and receives details

[1719] The device receives the recipe selected by the user and sends that information to the server. The server generates detailed information about the selected recipe and sends it to the device. The detailed information includes cooking instructions, a list of ingredients, calorie information, and nutritional balance. The device displays this information on the user's screen.

[1720] Users provide feedback

[1721] After trying the suggested recipes, users can provide feedback from within the app, such as "It was delicious" or "I wish it was a little saltier." The device then sends this feedback information to the server.

[1722] The server collects feedback and improves the algorithm

[1723] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[1724] In this way, by combining emotion recognition technology, the system can quickly suggest optimal recipes based on the user's current emotional state, significantly reducing the effort and stress of cooking dinner. As a specific example, a user can select a recipe based on the items in the refrigerator and their own emotional state, and then actually recreate it at home, saving time and effort.

[1725] The processing flow will be explained below.

[1726] Step 1:

[1727] The user launches the "Dinner Omakase" app on their smartphone and uses the app's camera function to take a picture of the contents of the refrigerator. The device temporarily saves the image as a "refrigerator image."

[1728] Step 2:

[1729] The terminal sends the "refrigerator interior image" to the server, and the image data is transferred to the server via the network.

[1730] Step 3:

[1731] The server receives the "refrigerator image" sent by the user. The received image data is passed to the image recognition module in the server.

[1732] Step 4:

[1733] The server's image recognition module uses deep learning-based algorithms to analyze and identify food items in the image, such as "tomato," "chicken," and "broccoli."

[1734] Step 5:

[1735] The server generates a list of the identified food items and stores it in a database. The identified food item information is stored in the database.

[1736] Step 6:

[1737] If the user is using a device with a camera, the device captures the user's facial expression data and sends the data to an emotion recognition engine.

[1738] Step 7:

[1739] The server's emotion recognition engine analyzes the facial expression data and identifies the user's emotional state (e.g., "satisfied," "fatigued," "stressed," etc.) This emotional state information is stored in a database.

[1740] Step 8:

[1741] The server searches the recipe database based on the food item list and emotion recognition results to retrieve related recipes. Multiple recipes are found in the recipe database.

[1742] Step 9:

[1743] The server prioritizes the retrieved recipes based on the user's past selection history, preferences, and current emotional state. For example, if the user is "fatigued," recipes that are easy to prepare and can be made quickly will be prioritized.

[1744] Step 10:

[1745] The server generates a prioritized recipe list and transmits it to the user's terminal, where it is displayed in a format that the user can view.

[1746] Step 11:

[1747] The terminal will display a message to the user similar to:

[1748] We recommend the following recipes for dinner tonight:

[1749] 1. Grilled Chicken and Broccoli (Cooking time: 20 minutes)

[1750] 2. Tomato and Chicken Pasta (Cooking time: 15 minutes)

[1751] Which one would you like?

[1752] The user selects one of the suggested recipes.

[1753] Step 12:

[1754] The terminal receives the recipe selected by the user and transmits the information to the server.

[1755] Step 13:

[1756] The server generates detailed information about the selected recipe and sends it to the device, including cooking instructions, a list of ingredients, calorie information, and nutritional balance.

[1757] Step 14:

[1758] The device displays the detailed information received from the server to the user, allowing the user to check the specific cooking steps and ingredient information for the selected recipe.

[1759] Step 15:

[1760] After the user tries the recipe, they submit feedback from within the app. The device collects the user's feedback information and sends it to the server.

[1761] Step 16:

[1762] The server stores the received feedback in a database, analyzes the stored feedback information, and updates the recommendation algorithm to improve the accuracy of future recipe suggestions.

[1763] Example 2

[1764] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1765] Conventional food management and recipe recommendation systems have difficulty not only accurately identifying the food in a user's refrigerator, but also automating optimal recipe recommendations that take into account the user's emotional state. This has left users unable to quickly choose an appropriate dish that suits their current mood while utilizing the food in their refrigerator, often resulting in stress when preparing dinner. To solve this problem, a recipe recommendation system that simultaneously considers the food in the refrigerator and the user's emotional state is needed.

[1766] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1767] In this invention, the server includes means for allowing a user to acquire an image of the contents of the refrigerator, image recognition means for identifying the acquired image and specifying food items, emotion recognition means for acquiring data on the user's facial expression and analyzing the data to identify the user's emotional state, means for suggesting recipes based on information on the identified food items and the user's emotional state, and means for providing detailed information on the suggested recipes. This makes it possible to quickly suggest optimal recipes based on information on the food items in the refrigerator and the user's current emotional state.

[1768] "User" refers to an individual who uses the system to scan the contents of their refrigerator and receive recipe suggestions.

[1769] "Means of acquiring images" refers to the process in which a user takes a photo of the food in the refrigerator using the camera function of a smartphone or other device.

[1770] "Image recognition means" refers to technology for analyzing acquired image data and identifying food items in the refrigerator.

[1771] "Facial expression data" refers to image data captured by a camera of a user's face, and is used for emotion recognition.

[1772] "Emotion recognition means" refers to technology that analyzes facial expression data and identifies the user's emotional state (e.g., "satisfaction," "fatigue," "stress," etc.).

[1773] "Means for suggesting recipes" refers to a function that selects and suggests appropriate recipes based on information about the identified food items and the user's emotional state.

[1774] "Means for providing detailed information" refers to a function that provides users with detailed information such as cooking instructions for the suggested recipe, a list of required ingredients, calorie information, and nutritional balance.

[1775] The present invention is a system that allows a user to scan the contents of a refrigerator using a smartphone or tablet, and combines image recognition technology and emotion recognition technology to identify food items and suggest optimal recipes based on the user's emotional state. Next, a specific embodiment of this system will be described.

[1776] The system uses the following hardware and software:

[1777] Mobile devices such as smartphones and tablets

[1778] Camera on the device

[1779] Deep learning-based image recognition modules (e.g., TensorFlow and PyTorch)

[1780] Emotion recognition engine (e.g., Microsoft Azure Face API)

[1781] Recipe Database

[1782] Specific steps

[1783] User scans the contents of the refrigerator

[1784] The user launches the "Dinner Omakase" app installed on their smartphone and uses the camera to take a picture of the contents of the refrigerator. The device acquires the captured image data and sends it to the server in real time.

[1785] The server performs image recognition

[1786] The server receives the image of the refrigerator sent by the user and uses a deep learning-based image recognition module to recognize food items in the image, such as "tomato," "chicken," and "broccoli," and stores this information in a database.

[1787] The server performs emotion recognition.

[1788] When a user is using a device with a camera, the device captures the user's facial expression data and sends it to an emotion recognition engine in the server. The server analyzes the facial expression data and identifies the user's emotional state (e.g., "satisfied," "fatigued," "stressed," etc.).

[1789] Server suggests recipes

[1790] The server searches the recipe database based on the identified food item list and the emotion recognition results to retrieve relevant recipes. For example, if the user is in a "fatigue" state, it will prioritize recipes that are easy to make and can be made in a short time.

[1791] The device notifies the user of recipe suggestions

[1792] The server generates a list of suggested recipes and sends it to the user's device, where the app displays the list on the screen and provides specific lists and options.

[1793] User selects a recipe and receives more information

[1794] The user selects one of the suggested recipes. The selected recipe information is sent from the device to the server, and the server generates detailed cooking instructions, a list of ingredients, calorie information, nutritional balance, and other information and sends it to the device. The device then displays this information on the user's screen.

[1795] Users provide feedback

[1796] After trying the suggested recipes, users provide feedback from within the app, which the device then captures and sends to the server.

[1797] The server accumulates feedback and improves the algorithm

[1798] The server accumulates the feedback sent by users in a database and reflects it in future recipe suggestions. The server analyzes the feedback data and improves the recipe recommendation algorithm to achieve more accurate recipe suggestions.

[1799] Examples of prompt statements

[1800] For example, if a user uses a "Dinner Picker" app to scan the items "clams" and "garlic" in the refrigerator, and the app recognizes the user's facial expression as indicating a "fatigue" state, the prompt sentence based on this example might be:

[1801] Use the "Dinner Omakase" app to scan the contents of your refrigerator (clams, garlic) and suggest the best dinner recipe through image and emotion recognition. The user's emotional state is "fatigue."

[1802] In this way, by combining emotion recognition technology, the system can quickly suggest optimal recipes based on the user's current emotional state, significantly reducing the effort and stress of cooking dinner.

[1803] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1804] Processing flow

[1805] Step 1:

[1806] The user launches the app installed on their smartphone and takes a picture of the contents of the refrigerator. At this time, the user uses the camera function in the app to capture image data (images of the food in the refrigerator). The image data obtained as input is saved in the device's internal storage. Specifically, the user taps the camera icon on the app screen and takes a picture of the inside of the refrigerator with the smartphone's camera.

[1807] Step 2:

[1808] The device sends the acquired image data to the server. At this time, the device's app uploads the image data to the server via an Internet connection. The input is the image data acquired in step 1, and the output is the image data sent to the server. Specifically, the device uploads the data to the server using a background process.

[1809] Step 3:

[1810] The server receives image data sent by the user and identifies food items using an image recognition module. The image data received as input is analyzed using a deep learning-based image recognition algorithm (e.g., TensorFlow or PyTorch) to generate a list of food items. Specifically, the server calls the image recognition API and generates a list of items such as "tomato," "chicken," and "broccoli."

[1811] Step 4:

[1812] The server stores the identified food item list in a database, where the input is the food item list obtained in step 3 and the output is the information stored in the database. Specifically, the server opens a database connection and executes an SQL query to store the listed food items.

[1813] Step 5:

[1814] When a user is using a device with a camera, the device acquires the user's facial expression data and sends it to the emotion recognition engine. The input is the user's facial expression image data, and the output is the transmission of facial expression data from the device to the server. Specifically, the user shows an expression in front of the camera, and the device captures it and uploads it to the server.

[1815] Step 6:

[1816] The server analyzes the received facial expression data using an emotion recognition engine to identify the user's emotional state. The input is facial expression data, and the output is the identified emotional state (e.g., "satisfied," "fatigued," "stressed," etc.). Specifically, the server calls an emotion recognition API (e.g., Microsoft Azure Face API) to analyze the emotional state.

[1817] Step 7:

[1818] The server searches the recipe database based on the identified food item list and emotion recognition results to suggest optimal recipes. The input is the food item list and the emotional state, and the output is a list of related recipes. Specifically, the server generates a query and executes an SQL query to retrieve suitable recipes from the recipe database.

[1819] Step 8:

[1820] The server generates a list of suggested recipes and sends it to the user's device. The input is the recipe list, and the output is sending the recipe list to the device. Specifically, the server sends the generated recipe list to the device via push notification.

[1821] Step 9:

[1822] The device displays the suggested recipe list on the user's screen. The input is the recipe list sent from the server, and the output is the list displayed on the user's screen. In concrete terms, the app receives the recipe list and displays it using a UI component.

[1823] Step 10:

[1824] The user selects one of the suggested recipes. The input is the displayed recipe list, and the output is the selected recipe information. Specifically, the user selects a recipe by tapping on the screen.

[1825] Step 11:

[1826] The terminal sends the selected recipe information to the server. The input is the recipe information selected by the user, and the output is the sending of a recipe list to the server. In concrete terms, the terminal uploads the selected content to the server.

[1827] Step 12:

[1828] The server generates detailed information for the selected recipe and sends it to the terminal. The input is the selected recipe information, and the output is the detailed recipe information. Specifically, the server generates detailed cooking instructions and ingredient lists and sends them to the terminal.

[1829] Step 13:

[1830] The device displays detailed recipe information on the user's screen. The input is the detailed recipe information sent from the server, and the output is the detailed information displayed on the user's screen. Specifically, the app receives the detailed information and displays it as a description or list.

[1831] Step 14:

[1832] The user enters feedback within the app. The input is the user's feedback comment, and the output is saving the feedback information to the device. Specifically, the user writes a comment in the feedback form and taps the submit button.

[1833] Step 15:

[1834] The terminal sends feedback information to the server. The input is the feedback information provided by the user, and the output is the transmission of the feedback information to the server. In concrete terms, the terminal uploads the transmitted feedback to the server.

[1835] Step 16:

[1836] The server accumulates the feedback sent by the user in a database and reflects it in future recipe suggestions. The input is the feedback data, and the output is an updated recipe suggestion algorithm. Specifically, the server stores the feedback data in storage and uses it to improve the machine learning algorithm.

[1837] (Application example 2)

[1838] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1839] Conventional recipe suggestion systems could suggest recipes based on the contents of a user's refrigerator, but they could not take the user's emotional state into account. This made it difficult for users to find products and recipes that matched their emotions at any given time. The present invention aims to solve these problems by providing a system that detects a user's emotional state and suggests optimal products and recipes based on that emotional state.

[1840] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1841] In this invention, the server includes means for a user to acquire an image of the contents of the refrigerator, image recognition means for identifying food items by recognizing the acquired image, means for suggesting recipes based on information on the identified food items, means for providing detailed information on the suggested recipes, emotion recognition means for scanning the user's facial expression to identify their emotional state, means for recommending products based on the user's emotional state, and means for providing detailed information on the recommended products, thereby enabling the user to receive recommendations that are optimal for their emotional state at the time.

[1842] "User" refers to an individual who uses the system.

[1843] "Means for acquiring an image of the contents of the refrigerator" refers to a function that allows a user to take a picture of the contents of the refrigerator using the camera on a smartphone or tablet and acquire the image data.

[1844] "Image recognition means" refers to technology for analyzing captured image data and identifying food items present in the image.

[1845] "Food items" refer to ingredients or foods present in an image, including specific items such as tomatoes or chicken.

[1846] The "means for suggesting recipes" refers to a function that provides the user with appropriate cooking methods and menus based on information about food items identified by the image recognition means.

[1847] "Means for providing detailed information" refers to a function that presents the user with specific cooking steps, a list of ingredients, calorie information, etc. for a recipe.

[1848] "Emotion recognition means for scanning facial expressions to identify emotional states" refers to technology for analyzing a user's facial expression data and identifying their emotional state at that time (e.g., joy, fatigue, etc.).

[1849] "Means for recommending products" refers to a function that suggests appropriate products to users based on their emotional state and past behavioral history.

[1850] "Means for providing detailed information about recommended products" refers to a function that displays detailed information about recommended products, such as descriptions, prices, and reviews, to users.

[1851] The present invention relates to a system that allows users to scan the contents of their refrigerator using a smartphone or tablet, combines image recognition technology with emotion recognition technology to identify food items, and suggests optimal recipes and products based on the user's emotional state.

[1852] System configuration

[1853] The system consists of the following main components:

[1854] 1. User device: A device equipped with a camera and display, such as a smartphone or tablet.

[1855] 2. Server: A cloud service that performs image recognition, emotion recognition, and recipe and product suggestions.

[1856] 3. Database: Stores food items, emotional states, recipes, product information, user behavior history and feedback.

[1857] Program execution procedure

[1858] 1. User scans the contents of the refrigerator

[1859] The user launches the dedicated application and takes a picture of the contents of the refrigerator using the camera function of their smartphone or tablet. The device acquires the image data and sends it to the server.

[1860] 2. Image recognition by the server

[1861] The server receives the image of the refrigerator sent by the user. The server is equipped with a deep learning-based image recognition module, which recognizes food items in the image. For example, food items such as "tomato," "chicken," and "broccoli" are identified in the image. The list of identified food items is stored in a database.

[1862] 3. Emotion recognition

[1863] When a user uses a device with a camera, the device captures the user's facial expression data and sends it to the server. The emotion recognition engine in the server analyzes the facial expression data and identifies the user's emotional state. For example, emotional states such as "joy," "fatigue," and "stress" are analyzed.

[1864] 4. Recipe and product suggestions

[1865] The server searches the recipe and product database based on the identified food item list and emotion recognition results to retrieve related recipes and products. Related recipes and products are suggested based on the user's emotional state. For example, if the user is "fatigued," recipes that are easy to make in a short time and products that help them relax are suggested first.

[1866] 5. Notifications on user devices

[1867] The server generates suggested recipes and product lists and sends them to the user's device. The device app displays the suggested recipes and product lists on the user's screen and sends interactive messages. For example, the following message might be displayed:

[1868] Here are some products we recommend for you right now:

[1869] 1. Grilled Chicken and Broccoli (Cooking time: 20 minutes)

[1870] 2. Tomato and Chicken Pasta (Cooking time: 15 minutes)

[1871] 3. Stress Relief Ball

[1872] 4. Herbal tea

[1873] Which one would you like?

[1874] Hardware and software used

[1875] Hardware:

[1876] Smartphone or tablet camera: Used to capture images of the user's facial expressions and the inside of the refrigerator.

[1877] software:

[1878] Python: The implementation language for the entire program.

[1879] opencv-python: Used to process facial expression and fridge image data.

[1880] Pillow: A library for image processing.

[1881] numpy: Used to process data.

[1882] TensorFlow: Emotion recognition using pre-trained models.

[1883] Prompt Sentence Examples

[1884] Write code to analyze a user's emotional state using facial images and recommend products that best match their emotions. Use the camera function to capture facial images and use TensorFlow to recognize emotions with a pre-trained model. Write a set of Python code to display a list of products based on the recognized emotion.

[1885] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1886] Step 1:

[1887] The user launches an application on their smartphone or tablet and uses the camera function to take a picture of the contents of the refrigerator. The device acquires the captured image data (image of the inside of the refrigerator) and sends this image to the server. The input is the image data captured by the user, and the output is the transfer of the image data to the server.

[1888] Step 2:

[1889] The server receives an image of the inside of the refrigerator sent by the user. It uses a deep learning-based image recognition module to recognize food items in the image. The input is image data of the inside of the refrigerator, and the output is a list of identified food items. Specifically, the image data is preprocessed and input into a model to identify food items.

[1890] Step 3:

[1891] The identified food item list is saved in a database. The input is the food item list output from image recognition, and the output is the process of saving it to the database. Specifically, the type and quantity of food items are written to the database.

[1892] Step 4:

[1893] The user takes a photo of their own facial expression using the camera function of their smartphone or tablet. The device acquires the facial expression data and sends it to the server. The input is the facial expression image taken by the user, and the output is the transfer of the facial expression data to the server.

[1894] Step 5:

[1895] The server receives the facial expression images and analyzes the facial expression data using an emotion recognition engine. The input is the facial expression image data, and the output is the user's emotional state. Specifically, the facial expression data is preprocessed and input into an emotion recognition model to identify the emotional state.

[1896] Step 6:

[1897] The server searches the recipe and product databases based on the identified food item list and emotional state to retrieve related recipes and products. The input is the food item list and emotional state, and the output is a list of recommended recipes and products. Specifically, it extracts information that matches the conditions from the recipe database and product database.

[1898] Step 7:

[1899] The server generates a list of recommended recipes and products and sends it to the user's device. The input is the recommendation list, and the output is data transmission to the user's device. Specifically, the server formats the selected information and delivers it to the user's device.

[1900] Step 8:

[1901] The user device displays the suggested recipes and product list on the screen and presents interactive messages to the user. For example, it presents multiple options along with the message "Here are the products we recommend for you right now:" The input is the recommendation list data from the server, and the output is the display to the user. Specifically, the information is presented visually using a GUI.

[1902] Step 9:

[1903] The user checks the detailed information of the suggested recipes and products and selects a specific recipe or product. The input is the user's selection, and the output is the transmission of the selected data to the server. Specifically, the system detects the user's action of pressing the selection button and sends the selected information to the server.

[1904] Step 10:

[1905] The server receives the user's selection information, generates detailed information about the selected recipe or product, and sends it to the user's device. The input is the user's selection information, and the output is a response with detailed information. Specifically, the server retrieves detailed information from the database, formats it, and provides it to the user.

[1906] Step 11:

[1907] The user terminal displays detailed information on the screen and provides cooking instructions and product details. The input is the detailed information data sent from the server, and the output is the display to the user. Specifically, the detailed information is presented visually using a GUI.

[1908] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1909] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1910] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1911] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1912] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1913] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1914] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1915] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1916] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1917] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1918] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1919] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1920] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1921] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1922] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1923] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1924] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1925] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1926] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1927] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1928] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1929] The following is further disclosed regarding the above embodiment.

[1930] (Claim 1)

[1931] A means for a user to acquire an image of the contents of a refrigerator;

[1932] image recognition means for identifying captured images to identify food items;

[1933] a means for suggesting recipes based on the identified food item information;

[1934] a means for providing detailed information about the suggested recipe;

[1935] A system including:

[1936] (Claim 2)

[1937] 10. The system of claim 1, further comprising means for prioritizing recipes by taking into account a user's past selections and preferences.

[1938] (Claim 3)

[1939] 10. The system of claim 1, further comprising means for receiving feedback from a user to improve accuracy of the recipe suggestions.

[1940] "Example 1"

[1941] (Claim 1)

[1942] A means for a user to acquire an image of the contents of a refrigerator;

[1943] image recognition means for identifying captured images to identify food items;

[1944] a means for suggesting recipes based on the identified food item information;

[1945] a means for providing detailed information about the suggested recipe;

[1946] a means for users to provide feedback;

[1947] a means for accumulating the provided feedback to improve the recipe suggestion algorithm; and

[1948] A system including:

[1949] (Claim 2)

[1950] 10. The system of claim 1, further comprising means for prioritizing suggested recipes based on the user's past selections and preferences.

[1951] (Claim 3)

[1952] 2. The system according to claim 1, further comprising means for notifying a user terminal of the suggested recipe list.

[1953] "Application Example 1"

[1954] (Claim 1)

[1955] A means for a user to acquire an image of the contents of a refrigerator;

[1956] image recognition means for identifying captured images to identify food items;

[1957] a means for suggesting recipes based on the identified food item information;

[1958] a means for providing detailed information about the suggested recipe;

[1959] A means to automatically list the missing materials and order materials from a delivery service;

[1960] A system including:

[1961] (Claim 2)

[1962] 10. The system of claim 1, further comprising means for prioritizing recipes by taking into account a user's past selections and preferences.

[1963] (Claim 3)

[1964] 10. The system of claim 1, further comprising means for receiving feedback from a user to improve accuracy of the recipe suggestions.

[1965] "Example 2: Combining Emotion Engines"

[1966] (Claim 1)

[1967] A means for a user to acquire an image of the contents of a refrigerator;

[1968] image recognition means for identifying captured images to identify food items;

[1969] emotion recognition means for acquiring facial expression data of a user and analyzing the data to identify the user's emotional state;

[1970] means for suggesting recipes based on the identified food item information and the user's emotional state;

[1971] a means for providing detailed information about the suggested recipe;

[1972] ...

[1973] A system including:

[1974] (Claim 2)

[1975] 10. The system of claim 1, further comprising means for prioritizing recipes by taking into account a user's past selections and preferences.

[1976] (Claim 3)

[1977] 10. The system of claim 1, further comprising means for receiving feedback from a user to improve accuracy of the recipe suggestions.

[1978] "Application example 2 when combining emotion engines"

[1979] (Claim 1)

[1980] A means for a user to acquire an image of the contents of a refrigerator;

[1981] image recognition means for identifying captured images to identify food items;

[1982] a means for suggesting recipes based on the identified food item information;

[1983] a means for providing detailed information about the suggested recipe;

[1984] emotion recognition means for scanning the user's facial expressions to identify their emotional state;

[1985] means for recommending products based on the emotional state of a user;

[1986] a means for providing detailed information about the recommended products;

[1987] A system including:

[1988] (Claim 2)

[1989] 10. The system of claim 1, further comprising means for prioritizing recipes by taking into account a user's past selections and preferences.

[1990] (Claim 3)

[1991] 10. The system of claim 1, further comprising means for receiving feedback from the user to improve accuracy of the recipe and product suggestions. [Explanation of symbols]

[1992] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for a user to acquire an image of the contents of a refrigerator; image recognition means for identifying captured images to identify food items; a means for suggesting recipes based on the identified food item information; a means for providing detailed information about the suggested recipe; A system including:

2. 10. The system of claim 1, further comprising means for prioritizing recipes by taking into account a user's past selections and preferences.

3. The system of claim 1 further comprising means for receiving feedback from a user to improve accuracy of the recipe suggestions.

Images