System

Abstract

A system is provided.SOLUTION: The system includes means for generating a recipe based on the obtained food list and preference information, means for recognizing and identifying the food in the refrigerator by image analysis, means for personalizing the recipe based on the user's preferences with reference to historical usage data, means for automatically generating and sending a delivery request if the required food and utensils are missing, and means for providing the recipe and cooking procedure to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many households and individuals often find themselves unsure of how to use leftover ingredients in their refrigerators, or find their daily cooking routines becoming monotonous. This not only wastes ingredients, but also wastes money on food. It is also difficult to prepare healthy and varied meals when you are busy. To solve these problems, a system is needed that allows users to efficiently use the ingredients they have and easily find a wide variety of recipes. [Means for solving the problem]

[0005] The present invention provides a system including a means for generating recipes based on an acquired food list and preference information, a means for identifying and identifying foods in a refrigerator through image analysis, a means for referencing past usage data and personalizing recipes based on a user's preferences, a means for automatically generating and sending a delivery request when necessary foods and utensils are in short supply, and a means for providing recipes and cooking instructions to a user.The system also solves the above-mentioned problems by including a means for making suggestions to minimize food waste by taking into account the expiration dates of foods and utensils, and a means for receiving and analyzing user input as text data and image data.

[0006] A "food list" is a list of ingredients that the user owns, and is information used to generate recipes.

[0007] "Preference information" refers to information about a user's tastes and preferences regarding ingredients and cooking, and is data used to personalize recipes.

[0008] The "recipe generator" is a function that automatically generates new recipes based on the food list and preference information.

[0009] "Image analysis" is a technology that analyzes images of a refrigerator and identifies and characterizes the ingredients contained within.

[0010] "Identification" is the process of classifying ingredients identified through image analysis into specific categories and names.

[0011] "Past usage data" is a record of recipes and ingredients a user has previously used, and is information used to understand the user's preferences.

[0012] A "Delivery Request" is a request to order necessary food or equipment from an outside supply service if such items are in short supply.

[0013] "Personalization" means customizing recipes and suggestions based on a user's individual preferences and past usage data.

[0014] "Expiration date" is the date by which food or equipment can be used, and taking this into consideration can help minimize food waste.

[0015] "Text data" refers to data that includes text information entered by the user.

[0016] The "recipe and cooking procedure providing means" is a function that provides the user with the generated recipe and its cooking method in an easy-to-understand manner. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a system for solving the problems of cooking in a rut and food waste in busy daily lives. Below, an embodiment of this system is described.

[0039] 1. Getting User Input

[0040] To help users efficiently utilize the ingredients in their refrigerator and enjoy a wide variety of dishes, the device first acquires input data. The device provides users with the options to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their favorite dishes. This allows the device to collect information about the user.

[0041] 2. Image Analysis and Data Conversion

[0042] The image data of the refrigerator sent by the device is received by the server. The server uses an image analysis algorithm to identify and identify the ingredients present in the refrigerator. At the same time, the list of ingredients and cooking preferences are also analyzed and converted into text data. For example, ingredients such as "carrots," "bell peppers," "chicken," and "eggs" can be recognized from the image and recorded as text data.

[0043] 3. Recipe Creation and Search

[0044] The server generates recipes based on the acquired ingredient list and preference information. Using a generative AI model, it suggests recipes that make the most of the ingredients the user has. For example, if a user has chicken, eggs, carrots, and green peppers, it will suggest recipes such as oyakodon (chicken and egg rice bowl) or stir-fry. It also references an existing recipe database to search for and provide related recipe information.

[0045] 4. Personalizing your data

[0046] The server refers to the user's past usage data and personalizes recipes based on the user's preferences and habits. It also takes into account the expiration dates of ingredients and makes suggestions to minimize food waste. For example, it can prevent food waste by suggesting recipes that prioritize ingredients with a short expiration date.

[0047] 5. Delivery of missing ingredients and equipment

[0048] If the server does not have the ingredients or equipment required for the generated recipe on hand, it automatically generates a delivery request and sends an order to an external supply service. The delivery service quickly arranges for the missing ingredients and equipment and delivers them to the user. For example, if there is a shortage of "chicken," that information is sent to the delivery service and arrangements are made to procure it.

[0049] 6. Results display and cooking assistance

[0050] The server sends the final recipe data, cooking instructions, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner to the user, providing text and visual content to guide the user through the cooking methods and procedures. For example, the device displays detailed instructions, cooking time, and a list of necessary equipment for making "Oyakodon."

[0051] Specific examples

[0052] Example 1: Upload an image of a refrigerator

[0053] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app.

[0054] 2. The server analyzes the image and recognizes "carrots," "bell peppers," "chicken," and "eggs."

[0055] 3. The server generates a recipe for "Oyakodon" based on these ingredients and sends the cooking instructions to the terminal.

[0056] 4. The device displays cooking instructions to the user.

[0057] Example 2: Enter a list of ingredients

[0058] 1. A user types "tomato, cheese, pasta" into a text box.

[0059] 2. The server analyzes this, searches for related recipes, and suggests "pasta with tomatoes and cheese."

[0060] 3. The server generates a recipe that takes expiration dates into account and sends the details to the device.

[0061] 4. The device displays the recipe and cooking instructions to the user.

[0062] In this way, the present invention can effectively utilize leftover ingredients in the refrigerator, preventing daily cooking from becoming monotonous and providing users with a fresh and healthy diet.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] A user launches the application using a device such as a smartphone or tablet, which then provides the user with the option to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their preferred dish.

[0066] Step 2:

[0067] Users can take a picture of their refrigerator and upload it to the application, or enter their ingredient list and cooking preferences into a text box.

[0068] Step 3:

[0069] The terminal sends the user's input data (image data or text data) to the server.

[0070] Step 4:

[0071] The server uses an image analysis algorithm to analyze the image data received from the refrigerator. Specifically, it applies a food recognition model to identify ingredients from the image. This allows ingredients such as "carrots," "bell peppers," "chicken," and "eggs" to be identified and classified.

[0072] Step 5:

[0073] The server performs text analysis on the user's input of ingredients and cooking preferences to extract ingredients and cooking genres, which are also used to generate recipes.

[0074] Step 6:

[0075] The server uses the generative AI model to generate customized recipes based on the user's food list and preferences, and also references existing recipe databases such as Classy to search and retrieve related recipe information.

[0076] Step 7:

[0077] The server references the user's past usage data, taking into account information such as past favorite recipes and commonly used ingredients, and personalizes recipes based on the user's preferences.

[0078] Step 8:

[0079] The server creates a list of ingredients and equipment needed for the generated recipe, checking the user's ingredient list to ensure all ingredients are available.

[0080] Step 9:

[0081] The server adjusts recipes taking into account the expiration dates of ingredients, suggesting recipes that prioritize ingredients with close expiration dates and preventing food waste.

[0082] Step 10:

[0083] If the server is running low on ingredients or equipment, it automatically generates a delivery request, which is then sent to an external supply service (e.g., an online shopping site).

[0084] Step 11:

[0085] The server receives confirmation from the delivery service and sends a notification to the user, for example, "Your chicken will arrive in 30 minutes."

[0086] Step 12:

[0087] The server sends the final recipe data, cooking instructions, and information on missing ingredients and utensils to the terminal.

[0088] Step 13:

[0089] The device receives information and displays it to the user in an easy-to-understand manner, such as providing recipe steps, necessary equipment, and links to cooking videos.

[0090] Step 14:

[0091] The user begins cooking according to the recipe displayed on the device, with the help of a timer and additional tips provided by the app.

[0092] Step 15:

[0093] The device records the user's cooking progress, sends the data back to the server, and stores it in a database for future personalization.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] In today's busy lifestyles, cooking at home can easily become monotonous, and food in the refrigerator is often underutilized and thrown away. While this problem differs from household to household, the following issues need to be resolved: First, to efficiently utilize the food in the refrigerator and provide nutritionally balanced meals. Second, to suggest personalized recipes that take into account the user's preferences and past cooking history. Third, to assist the user in cooking by quickly arranging for ingredients and utensils that are in short supply.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes means for generating recipes based on the acquired ingredient list and preference information, means for identifying and identifying ingredients in the refrigerator through image analysis, means for referencing past usage data and personalizing recipes based on the user's preferences, means for automatically generating and sending a delivery request when necessary ingredients and cooking utensils are in short supply, means for providing the user with recipes and cooking instructions, means for generating recipes based on input data using a generative AI model, and means for automatically generating prompt statements, inputting them into the AI ​​model, and generating recipes based on the results. This makes it possible to efficiently use ingredients in the refrigerator, provide personalized recipes that suit the user's preferences, and minimize food waste.

[0099] The "obtained ingredient list" refers to the information about ingredients in the refrigerator entered by the user, including image analysis results and text input data.

[0100] "Preference information" is data about a user's preferences based on their cooking preferences, past usage history, etc.

[0101] The "means for generating recipes" is a function that creates new recipes using a generative AI model and recipe database based on the acquired ingredient list and preference information.

[0102] "Image analysis" is a technique that processes image data to identify ingredients in the refrigerator. It includes image recognition algorithms and machine learning models.

[0103] "Means for identification" refers to the function of identifying and specifying ingredients based on certain criteria from the information obtained by image analysis.

[0104] "Past Usage Data" refers to all data from a user's previous use of the system, including information about recipes selected and ingredients used.

[0105] "Personalization" refers to a function that suggests individually appropriate recipes based on the user's past usage data and preferences.

[0106] "Delivery Request" refers to a request to send an order to an external supply service when necessary ingredients or cooking equipment are in short supply.

[0107] "Means for providing cooking instructions" is a function that clearly provides specific steps and information on necessary equipment when a user cooks a dish.

[0108] A "generative AI model" is an artificial intelligence model used for complex data processing such as image analysis and recipe generation, as well as natural language processing.

[0109] A "prompt sentence" is a text sentence that expresses a question or request that is input to a generative AI model, and the AI ​​generates a response based on this.

[0110] The present invention is a system that solves the problems of cooking in a rut and food waste in busy daily lives. This system efficiently utilizes ingredients in the refrigerator and suggests a wide variety of dishes, providing users with a fresh and healthy diet. Detailed embodiments for implementing this system are described below.

[0111] Getting User Input

[0112] Users use a dedicated application on their smartphone or PC to input information about the ingredients in their refrigerator. The input method is as follows:

[0113] 1. Take a picture of the ingredients in your refrigerator and upload it.

[0114] 2. Enter the ingredients list in the text box.

[0115] 3. Use the drop-down menu to select your preferred cuisine.

[0116] Image analysis and data conversion

[0117] The device sends image data, text data, or data on the user's favorite dishes to the server. The server then uses an image analysis algorithm (using, for example, OpenCV or TensorFlow) to identify the ingredients in the refrigerator. For example, it identifies "carrots," "bell peppers," "chicken," and "eggs" from the image and saves them as text data. This allows the server to obtain a list of ingredients and the user's favorite dishes.

[0118] Recipe Generation

[0119] The server generates a recipe using a generative AI model (for example, GPT-3 or a similar natural language processing model) based on the acquired ingredient list and preference information. Specifically, the server automatically generates a prompt sentence and inputs it into the generative AI model. For example, a prompt sentence such as "What recipes use chicken, eggs, carrots, and green peppers?" is generated and input into the AI ​​model. The recipe is generated based on the results obtained from the AI ​​model.

[0120] Data Personalization

[0121] The server refers to the user's past usage data and proposes personalized recipes that take into account the user's preferences and the expiration dates of ingredients, thereby prioritizing the use of ingredients with an approaching expiration date, thereby minimizing food waste.

[0122] Delivery of missing ingredients and cooking equipment

[0123] If the server is running low on ingredients or cooking utensils required for the generated recipe, it automatically generates a delivery request and sends it to an external supply service. For example, if there is a shortage of chicken, the server sends that information to the delivery service and quickly arranges for the missing ingredients or cooking utensils.

[0124] Result display and cooking assistance

[0125] The server sends the final recipe data, cooking instructions, and information on missing ingredients and cooking utensils to the device. The device then displays this information in an easy-to-understand manner to the user, guiding them through the cooking methods and procedures. For example, the device displays detailed instructions for making an "oyakodon," cooking time, and a list of necessary utensils. It also displays text and visual content (images and videos) to guide the user through the cooking process.

[0126] Specific examples

[0127] Example 1: Upload an image of a refrigerator

[0128] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app.

[0129] 2. The server analyzes the image and identifies "carrots," "bell peppers," "chicken," and "eggs."

[0130] 3. The server generates a recipe for "Oyakodon" based on these ingredients and sends the cooking instructions to the terminal.

[0131] 4. The device displays cooking instructions to the user.

[0132] Example 2: Enter a list of ingredients

[0133] 1. A user types "tomato, cheese, pasta" into a text box.

[0134] 2. The server analyzes this, searches for related recipes, and suggests "pasta with tomatoes and cheese."

[0135] 3. The server generates a recipe that takes expiration dates into account and sends the details to the device.

[0136] 4. The device displays the recipe and cooking instructions to the user.

[0137] In this way, users can efficiently utilize the ingredients in their refrigerator and enjoy fresh and healthy meals without getting bored with their daily cooking routine.

[0138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0139] Step 1: Getting User Input

[0140] Users launch the application on their smartphone or PC and enter information about the ingredients in their refrigerator by taking a picture of the refrigerator and uploading it, entering the list of ingredients in a text box, or selecting the option to choose their favorite dish.

[0141] Input: Image data of the contents of the refrigerator, text data, or favorite food information

[0142] Output: User input data (image data, text data, favorite food information)

[0143] Step 2: Image analysis and data conversion

[0144] The terminal transmits the acquired user input data to the server.

[0145] The server processes the received image data using an image analysis algorithm (e.g., OpenCV or TensorFlow) to identify the ingredients in the refrigerator. For example, it identifies "carrots," "bell peppers," "chicken," and "eggs" from the image.

[0146] Input: User-entered data (image data, text data, favorite food information)

[0147] Data processing: Identification and classification of ingredients using image analysis

[0148] Output: Identified ingredients list (text data)

[0149] Step 3: Generate the recipe

[0150] Based on the ingredient list and preference information acquired by the server, a recipe is generated using a generative AI model (e.g., GPT-3). The server generates a prompt sentence and inputs it into the generative AI model. For example, a prompt sentence such as "What is a recipe that uses chicken, eggs, carrots, and green peppers?" is generated and input into the AI ​​model.

[0151] Input: Ingredient list, preference information

[0152] Data processing: Recipe generation using generative AI models

[0153] Output: Suggested recipe (text data)

[0154] Step 4: Personalize your data

[0155] The server references the user's past usage data and generates personalized recipes that take into account the user's preferences and the expiration dates of ingredients. Specifically, it suggests recipes that prioritize ingredients with an approaching expiration date.

[0156] Input: Ingredient list, preference information, past usage data

[0157] Data processing: Consideration of preference analysis and expiration date data

[0158] Output: Personalized recipe (text data)

[0159] Step 5: Delivery of missing ingredients and cooking equipment

[0160] The server creates a list of ingredients and cooking tools needed based on the recipe. If any of the ingredients are missing, it automatically generates a delivery request and sends it to an external supply service. For example, if chicken is in short supply, it requests a delivery service to procure it.

[0161] Input: Personalized recipe, information about ingredients and cooking equipment the user has on hand

[0162] Data processing: Identifying missing ingredients and cooking utensils, generating automatic delivery requests

[0163] Output: Delivery request

[0164] Step 6: Displaying results and cooking assistance

[0165] The server sends the final recipe data, along with information on missing ingredients and cooking utensils, to the device. The device then displays this information in an easy-to-understand manner to the user, guiding them through the cooking method and steps. For example, it displays detailed instructions, cooking time, and a list of necessary utensils for making "Oyakodon."

[0166] Input: Final recipe data, cooking instructions, missing ingredients and utensils information

[0167] Data processing: Format conversion of cooking instructions and necessary information

[0168] Output: Cooking guide display for the user (text and visual content)

[0169] Through these steps, the system can efficiently utilize the ingredients in the refrigerator and provide users with a fresh and healthy diet.

[0170] (Application example 1)

[0171] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0172] Conventional food management systems do not adequately suggest recipes that efficiently utilize ingredients in the refrigerator, resulting in food waste. Furthermore, recipe suggestions that take into account the user's preferences and past cooking history are limited. In particular, the lack of a convenient way to provide cooking instructions or delivery of missing ingredients in busy daily lives has been a major problem for users.

[0173] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0174] In this invention, the server includes: means for generating recipes based on the acquired food list and preference information; means for identifying and recognizing foods in the refrigerator through image analysis; means for referencing past usage data and personalizing recipes based on the user's preferences; means for automatically generating and sending delivery requests when necessary foods and utensils are missing; means for providing recipes and cooking instructions to the user; means for processing food images taken with a smartphone; means for automatically recognizing the acquired food list using an image analysis algorithm; means for using a generative AI model to generate and suggest optimal recipes based on the acquired food list; and means for transmitting the missing list to a delivery service. This enables effective use of ingredients in the refrigerator, reduction of food waste, personalized recipe suggestions for users, and easy delivery ordering.

[0175] - "Food List" refers to a list of ingredients owned by the user.

[0176] "Preference information" refers to information about the cuisine and taste preferences of a user that has been selected in the past.

[0177] The "means for generating recipes" is a function that automatically creates recipes showing cooking methods and ingredients based on the acquired food list and preference information.

[0178] "Image analysis" refers to techniques and processes that process digital image data to extract specific information.

[0179] "Means for identifying and discerning food in a refrigerator" refers to a technology that uses image analysis to recognize and identify food items present in a refrigerator.

[0180] "Past usage data" refers to historical information such as ingredients used by the user, recipes selected, and ingredients ordered.

[0181] "Means for personalizing recipes based on user preferences" refers to technology that references past usage data and suggests recipes tailored to the user's specific preferences.

[0182] "Means for automatically generating and sending delivery requests" refers to a function that automatically detects when necessary food or equipment is missing and sends an order to an external delivery service.

[0183] "Means for providing recipes and cooking instructions" refers to a function that displays or communicates suggested recipes and their cooking instructions to the user.

[0184] The "means for processing food images taken by a smartphone" refers to the process of receiving image data taken by a user, analyzing it, and extracting a list of foods to be used.

[0185] An "image analysis algorithm" is a set of mathematical models or programs that process image data to recognize specific objects or text information.

[0186] A "generative AI model" is an artificial intelligence model that uses machine learning to generate new recipes and information from data.

[0187] A "delivery service" is a service organization that delivers ingredients and equipment to users.

[0188] The present invention is a system that efficiently utilizes ingredients in a refrigerator, provides a wide variety of dishes, and prevents food waste. This system starts when a user takes a photo of the ingredients in the refrigerator with their smartphone and uploads it to an application.

[0189] Hardware and software used

[0190] Hardware:

[0191] Smartphone: Taking pictures of ingredients and using applications.

[0192] Server: Image analysis, recipe generation, and delivery request processing.

[0193] Delivery service computer system: Arranges delivery of ingredients and cooking equipment.

[0194] software:

[0195] Flask: A web application framework in Python.

[0196] PIL (Pillow): Image processing library.

[0197] food_recognition: Food recognition algorithm.

[0198] recipe_generation: Recipe generation algorithm.

[0199] delivery_service: Third-party delivery service API.

[0200] Processing flow

[0201] 1. Getting User Input

[0202] Users take a photo of the ingredients in their refrigerator with their smartphone and upload the image to the application.

[0203] Alternatively, you're given the option to manually enter the ingredient list into a text box.

[0204] Information about the user's preferences and past cooking tastes can also be entered.

[0205] 2. Image Analysis and Data Conversion

[0206] The server receives the image data sent from the smartphone and reads the image data using Pillow.

[0207] Ingredients are identified using an image analysis algorithm (food_recognition), and the information is converted into text data.

[0208] For example, "tomato," "cheese," and "beef" can be identified from an image of a refrigerator and recorded as text data.

[0209] 3. Recipe Generation

[0210] A generative AI model (recipe_generation) is used to generate recipes based on the acquired ingredient list and user preference information.

[0211] For example, we suggest a recipe for Bolognese pasta using tomatoes, cheese, and beef.

[0212] 4. Creating a delivery request

[0213] The server checks the ingredients and equipment required for the generated recipe, and if any are missing, it automatically generates a delivery request for them.

[0214] It calls the delivery service's API and arranges for the necessary ingredients and equipment to be delivered to the user.

[0215] 5. Results display and cooking assistance

[0216] The server sends the final recipe data, cooking instructions, and information on missing ingredients and utensils to the smartphone.

[0217] The smartphone displays this information in an easy-to-understand way for the user, guiding them through the cooking process with visual and textual content.

[0218] Specific examples

[0219] Example 1: Uploading food images

[0220] Users take a picture of the inside of their refrigerator with their smartphone and upload it to the app.

[0221] The server analyzes the image and recognizes "carrots," "bell peppers," "chicken," and "eggs."

[0222] The server uses a generative AI model to generate a recipe for "Oyakodon" based on these ingredients and sends the cooking instructions to a smartphone.

[0223] The smartphone displays cooking instructions to the user.

[0224] Example 2: Ordering delivery of ingredients that are in short supply

[0225] A user uploads a picture of their refrigerator and discovers that they are out of "chicken."

[0226] The server generates a delivery request and places an order for "chicken" through the delivery service API.

[0227] The delivery service arranges for the chicken to be delivered to the user.

[0228] Example 3: Example of a prompt statement

[0229] Prompt: "How can I build a system that allows you to upload a picture of the ingredients in your refrigerator, suggest recipes based on the images, and order delivery for any missing ingredients? What are the basic features this system should have?"

[0230] As a result, this system can make effective use of ingredients in the refrigerator, reduce food waste, and provide users with personalized recipe suggestions and delivery services.

[0231] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0232] Step 1:

[0233] The user takes a photo of the food in the refrigerator with their smartphone and uploads the image to the application.

[0234] Specific actions

[0235] Users use their smartphone camera to take a picture of the ingredients in their refrigerator and upload the image to a dedicated app, which also offers the option to input user preference information and allows users to provide past usage data.

[0236] Input and Output

[0237] Input: Image data of ingredients, preference information

[0238] Output: Image data and preference information sent to the server

[0239] Step 2:

[0240] The server receives the image data and loads the image using Pillow.

[0241] Specific actions

[0242] The server receives the uploaded image data and reads the image using the Pillow image processing library, thereby obtaining the digital image data.

[0243] Input and Output

[0244] Input: Image data sent from a smartphone

[0245] Output: Loaded image data

[0246] Step 3:

[0247] The server uses an image analysis algorithm (food_recognition) to identify ingredients and convert them into text data.

[0248] Specific actions

[0249] The server analyzes the received image data using the food_recognition algorithm, identifies the ingredients in the image, converts the identified ingredients into text data, and creates a list.

[0250] Input and Output

[0251] Input: Image data

[0252] Output: Ingredient list (text data)

[0253] Step 4:

[0254] The server uses a generative AI model (recipe_generation) to generate a recipe based on the ingredient list and preference information.

[0255] Specific actions

[0256] The server uses a generative AI model to create an appropriate recipe based on the acquired ingredient list and the user's preferences. The generated recipe also includes specific cooking steps.

[0257] Input and Output

[0258] Input: Ingredient list, preference information

[0259] Output: Generated recipe and cooking instructions

[0260] Step 5:

[0261] The server checks for missing ingredients and equipment based on the generated recipe and generates a delivery request.

[0262] Specific actions

[0263] The server compares all ingredients included in the generated recipe with the user's current inventory, identifies any missing ingredients or utensils, and generates and sends a request to an external delivery service API.

[0264] Input and Output

[0265] Input: Generated recipe, ingredient list

[0266] Output: Delivery request

[0267] Step 6:

[0268] The server sends the final recipe data, cooking instructions, and delivery request results to the smartphone.

[0269] Specific actions

[0270] The server then sends the generated recipe data, cooking instructions, and delivery results for missing ingredients and utensils to a smartphone, where users can check the information using the smartphone app.

[0271] Input and Output

[0272] Input: Final recipe data, cooking instructions, delivery request results

[0273] Output: Recipe data and cooking instructions displayed on a smartphone

[0274] Through these steps, the system can effectively utilize the ingredients in the refrigerator and provide personalized recipe suggestions and delivery services.

[0275] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0276] The present invention is a system that efficiently utilizes ingredients and utensils that a user has, and also suggests recipes and cooking methods according to the user's emotional state. The following describes an embodiment of this system.

[0277] 1. Getting User Input

[0278] To help users efficiently utilize the ingredients in their refrigerator and enjoy a wide variety of dishes, the device first acquires input data. The device provides users with the option to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their preferred dishes. Additionally, the device also has the ability to collect voice and facial expression data to recognize the user's emotions.

[0279] 2. Image Analysis and Data Conversion

[0280] The image data of the refrigerator sent by the device is received by the server. The server uses an image analysis algorithm to identify and identify the ingredients present in the refrigerator. At the same time, the list of ingredients and cooking preferences are also analyzed and converted into text data. For example, ingredients such as "carrots," "bell peppers," "chicken," and "eggs" can be recognized from the image and recorded as text data.

[0281] 3. Emotional Recognition

[0282] The device collects the user's voice and facial expression data and sends it to the server. The server then uses an emotion engine to analyze and recognize the user's emotions from this data. For example, it analyzes changes in voice tone and facial expressions to determine whether the user is relaxed or stressed.

[0283] 4. Recipe Creation and Search

[0284] The server generates recipes based on the acquired ingredient list and preference information. Using a generative AI model, it suggests recipes that make the most of the ingredients the user has. It also references an existing recipe database to search for and provide related recipe information. For example, if a user has chicken, eggs, carrots, and green peppers, it will suggest recipes such as oyakodon (chicken and egg rice bowl) or stir-fry. Depending on the user's emotional state, it also prioritizes suggestions of very simple recipes and dishes with a relaxing effect.

[0285] 5. Personalization of Data

[0286] The server refers to the user's past usage data and personalizes recipes based on the user's preferences and habits. It also takes into account the expiration dates of ingredients and makes suggestions to minimize food waste. For example, it can prevent food waste by suggesting recipes that prioritize ingredients with a short expiration date.

[0287] 6. Delivery of missing ingredients and equipment

[0288] If the server does not have the ingredients or equipment required for the generated recipe on hand, it automatically generates a delivery request and sends an order to an external supply service. The delivery service quickly arranges for the missing ingredients and equipment and delivers them to the user. For example, if there is a shortage of "chicken," that information is sent to the delivery service and arrangements are made to procure it.

[0289] 7. Results display and cooking assistance

[0290] The server sends the final recipe data, cooking instructions, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner to the user, providing text and visual content to guide the user through the cooking methods and procedures. For example, the device may display detailed instructions, cooking time, and a list of necessary equipment for making "Oyakodon." Furthermore, the device may also provide relaxing music and video links depending on the user's emotional state.

[0291] Specific examples

[0292] Example 1: Uploading images and audio data of a refrigerator

[0293] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app, which also collects audio data.

[0294] 2. The server analyzes the image and recognizes "carrot," "bell pepper," "chicken," and "egg." At the same time, it analyzes the voice data and recognizes the user's emotional state.

[0295] 3. The server generates a recipe for "Oyakodon" based on this information and sends the cooking instructions to the device. If the user is feeling stressed, it also suggests relaxing music.

[0296] 4. The device displays and plays cooking instructions and relaxing music to the user.

[0297] Example 2: Input ingredients list and facial expression data

[0298] 1. The user enters "tomato, cheese, pasta" into the text box, and facial expression data is collected using the camera.

[0299] 2. The server analyzes this, searches for related recipes, and suggests "Tomato and Cheese Pasta." It also determines from facial expression data that the user is relaxed.

[0300] 3. The server generates a recipe that takes expiration dates into account and sends the details to the device.

[0301] 4. The device displays the recipe and cooking instructions to the user.

[0302] In this way, the present invention effectively utilizes leftover ingredients in the refrigerator and suggests recipes that take into consideration the user's emotional state, preventing cooking from becoming monotonous, reducing food waste, and providing a richer diet.

[0303] The processing flow will be explained below.

[0304] Step 1:

[0305] A user launches the application using a device such as a smartphone or tablet. The device provides the user with the options to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their preferred dish. It also prompts them to collect voice data and capture facial expression data.

[0306] Step 2:

[0307] Users take a picture of their refrigerator and upload it to the application, and are also asked to record a voice message or enter facial expression data via the camera to ascertain its emotional state.

[0308] Step 3:

[0309] The terminal sends the user's input data (image data, voice data, facial expression data, or text data) to the server.

[0310] Step 4:

[0311] The server analyzes the image data of the refrigerator it receives using an image analysis algorithm. Specifically, it uses a food recognition model to identify ingredients from the image. As a result, ingredients such as "carrots," "bell peppers," "chicken," and "eggs" in the refrigerator are identified.

[0312] Step 5:

[0313] The server analyzes the user's ingredient list and cooking preferences received as text data and extracts the ingredient list and cooking categories.

[0314] Step 6:

[0315] The server uses an emotion engine to analyze the transmitted voice and facial expression data and recognize the user's emotional state. For example, it can determine whether the user is relaxed or stressed based on changes in voice tone and facial expression.

[0316] Step 7:

[0317] The server generates recipes based on the acquired ingredient list and the user's recognized emotional state. Using a generative AI model, the server suggests recipes that make the most of the ingredients the user has. The server then adjusts the recipes according to the user's emotional state. For example, if the user is tired, the server suggests simple recipes that have a relaxing effect.

[0318] Step 8:

[0319] The server uses past usage data to personalize recipes based on the user's preferences and habits, and also takes into account expiration dates of ingredients, prioritizing recipes that minimize food waste.

[0320] Step 9:

[0321] The server creates a list of ingredients and tools required for the generated recipe, referencing the user's ingredient list to see what is available and what is missing.

[0322] Step 10:

[0323] If the server is missing ingredients or equipment, it automatically generates a delivery request, which is then sent to an external supply service.

[0324] Step 11:

[0325] The server receives confirmation from the delivery service and notifies the user when the missing ingredients or equipment will arrive, for example, "The chicken will arrive in 30 minutes."

[0326] Step 12:

[0327] The server sends the final recipe data, cooking instructions, and information on missing ingredients and utensils to the terminal.

[0328] Step 13:

[0329] The device displays the information received in an easy-to-understand manner to the user, providing recipe steps, necessary utensils, and links to cooking videos through text and visual content. It also displays relaxing music and videos based on the user's emotional state.

[0330] Step 14:

[0331] The user begins cooking by following the recipe displayed on the device, with the help of a timer and additional tips provided by the app.

[0332] Step 15:

[0333] The device records the user's cooking progress, sends the data back to the server, and stores it in a database for future personalization.

[0334] Example 2

[0335] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0336] Currently, many households are unable to efficiently utilize the ingredients in their refrigerators, resulting in food waste. It is also difficult for systems to suggest dishes that reflect the user's emotional state, which can reduce meal satisfaction. Furthermore, if the necessary ingredients or cooking utensils are in short supply, it takes time to procure them, making it difficult to start cooking quickly. There is a need to solve these issues and provide users with an efficient and satisfying cooking experience.

[0337] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0338] In this invention, the server includes means for receiving and analyzing user input as text data and image data, means for identifying and identifying foods in the refrigerator through image analysis, means for generating recipes based on the acquired food list and preference information, means for analyzing the user's emotional state using voice and facial expression data, means for suggesting recipes according to the user's emotional state, means for personalizing recipes based on the user's preferences by referencing past usage data, means for automatically generating and sending a delivery request when necessary foods and utensils are in short supply, means for providing the user with recipes and cooking instructions, and means for generating recipes using a generative AI model. This makes it possible to efficiently use ingredients in the refrigerator, suggest recipes according to the user's emotional state, and quickly procure the necessary ingredients and utensils.

[0339] "User input" refers to text data, image data, voice data, and facial expression data that a user provides to the system.

[0340] "Text data" refers to data entered by the user in the form of text, such as a list of ingredients or preference information.

[0341] "Image data" refers to images of food and other visual information in the refrigerator taken with a camera and uploaded to the system.

[0342] "Audio Data" refers to audio information collected for the purpose of analyzing a user's emotional state.

[0343] "Facial expression data" refers to emotional information extracted from images of a user's facial expressions captured by a camera.

[0344] "Image analysis" refers to the process of analyzing transmitted image data to identify and identify the food items in the refrigerator.

[0345] A "food list" refers to data that records in text format all the food in your refrigerator.

[0346] "Preference information" refers to information that represents a user's preferences for food and cooking.

[0347] "Emotional state" refers to the user's current psychological and emotional state, analyzed from their voice and facial expression data.

[0348] A "recipe" is a document or piece of information that lists the ingredients and steps needed to prepare a particular dish.

[0349] "Generative AI model" refers to an algorithmic model that uses artificial intelligence to generate new recipes.

[0350] "Past usage data" refers to data that records a user's system usage history.

[0351] "Food waste" refers to food that is discarded without being consumed.

[0352] "Delivery Request" means an order request sent to an external supply service to procure needed food and / or equipment.

[0353] "Personalization" refers to the process of providing users with optimized suggestions and services based on their past usage history and preferences.

[0354] The present invention is a system that efficiently utilizes the ingredients and cooking utensils that a user has, and also suggests recipes and cooking methods that correspond to the user's emotional state. Specific implementation methods of this system are described in detail below.

[0355] Getting User Input

[0356] Users input ingredient information and their favorite dishes using a smartphone or PC. The device provides users with the ability to take and upload images of their refrigerator, a text box for entering a list of ingredients, and check boxes and drop-down menus for selecting their favorite dishes. The device also collects user emotional data through voice input and facial recognition using a camera.

[0357] Image analysis and data conversion

[0358] The server receives the image data of the refrigerator sent by the device and uses image analysis algorithms (e.g., OpenCV or TensorFlow) to identify and distinguish the ingredients in the refrigerator. For example, it recognizes "carrots," "bell peppers," "chicken," and "eggs" from the image of the refrigerator and records them as text data. In addition, it analyzes the text list of ingredients and cooking preferences entered by the user and stores them in a database.

[0359] Emotion recognition

[0360] The device sends the collected voice and facial expression data to a server, which then analyzes it using an emotion analysis engine (e.g., IBM Watson or Microsoft Azure's emotion analysis API). This determines whether the user is relaxed or stressed. The analysis results are then recorded in a database.

[0361] Recipe creation and search

[0362] The server uses a generative AI model (e.g., GPT-4) to generate new recipes based on the acquired ingredient list and emotional data. For example, if a user has chicken, eggs, carrots, and green peppers, it will suggest recipes such as oyakodon (chicken and egg rice bowl) or stir-fry. It will also search an existing recipe database to provide related recipe information. If the user is feeling stressed, it will prioritize suggestions of easy recipes and dishes that have a relaxing effect.

[0363] Data Personalization

[0364] The server refers to the user's past usage data and personalizes recipes based on the user's preferences. For example, if there is an ingredient that is close to its expiration date, it can suggest a recipe that prioritizes using that ingredient, thereby reducing food waste.

[0365] Delivery of missing ingredients and equipment

[0366] The server checks whether the user has the ingredients and equipment required for the generated recipe. If they are in short supply, it automatically generates a delivery request and sends an order to an external supply service. For example, if there is a shortage of "chicken," that information is sent to the delivery service and arrangements are made for timely procurement.

[0367] Result display and cooking assistance

[0368] The server sends the final recipe data, cooking instructions, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner to the user, providing text and visual content to guide the user through the cooking method and steps. For example, it may display detailed instructions, cooking time, and a list of necessary equipment for making "Oyakodon." It may also provide relaxing music or video links depending on the user's emotional state.

[0369] Specific examples

[0370] Example 1: Uploading images and audio data of a refrigerator

[0371] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app, which also collects audio data.

[0372] 2. The device sends image and audio data to the server.

[0373] 3. The server analyzes the image and recognizes "carrot," "bell pepper," "chicken," and "egg." It also analyzes the voice data to understand the user's emotional state.

[0374] 4. The server generates a recipe for "Oyakodon" based on the recognized ingredients and emotional data. If it determines that the user is feeling stressed, it selects music with a relaxing effect.

[0375] 5. The server sends the generated recipe and music information to the device.

[0376] 6. The device displays and plays cooking instructions and relaxing music to the user.

[0377] Example 2: Input ingredients list and facial expression data

[0378] 1. The user enters "tomato, cheese, pasta" into the text box, and facial expression data is collected using the camera.

[0379] 2. The device sends the text and facial expression data to the server.

[0380] 3. The server analyzes the text data to search for relevant recipes and determines whether the user is relaxed based on facial expression data.

[0381] 4. The server generates a recipe for "Tomato and Cheese Pasta" and creates a recipe that takes into account the expiration date.

[0382] 5. The server sends the generated recipe to the device.

[0383] 6. The device displays the recipe and cooking instructions to the user.

[0384] In this way, the present invention effectively utilizes leftover ingredients in the refrigerator and suggests recipes that take into consideration the user's emotional state, preventing cooking from becoming monotonous, reducing food waste, and providing a richer diet.

[0385] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0386] Step 1: Getting User Input

[0387] Users open the app on their smartphone or PC and enter information about ingredients in their refrigerator and their favorite dishes, or take and upload images. Voice input and facial expression data from the camera are also collected. Image data, text data, voice data, and facial expression data are obtained as input, and this input data is sent to the system. The device collects this data, converts it into an appropriate format, and sends it to the server.

[0388] Step 2: Image analysis and data conversion

[0389] The server receives the image data of the refrigerator sent from the device and uses image analysis algorithms (e.g., OpenCV or TensorFlow) to identify the ingredients in the refrigerator. In this process, image data is input and converted into text data. For example, it identifies "carrots," "bell peppers," "chicken," and "eggs" from the image of the refrigerator and outputs them as text data. The server also analyzes the text list of ingredients and cooking preferences entered by the user and stores them in a database.

[0390] Step 3: Recognize emotions

[0391] The server receives the voice and facial expression data sent from the device and analyzes the user's emotional state using an emotion analysis engine (e.g., IBM Watson or Microsoft Azure's emotion analysis API). Voice and facial expression data are received as input, and they are analyzed to output information about the user's emotional state (relaxed, stressed, etc.). The analysis results are stored in a database.

[0392] Step 4: Creating and searching recipes

[0393] The server generates new recipes using a generative AI model (e.g., GPT-4) based on the acquired ingredient list, emotional data, and preference information. In this process, the ingredient list and emotional state are input as text data, and recipe suggestions based on them are output. For example, if a user has "chicken," "eggs," "carrots," and "green peppers," specific recipes such as "oyakodon" (chicken and egg rice bowl) or "stir-fry" are generated. The server also searches an existing recipe database and provides related recipe information. The preferred recipes change depending on the user's emotional state.

[0394] Step 5: Personalize your data

[0395] The server refers to the user's past usage data and personalizes recipes by taking into account the user's preferences, habits, and food expiration dates. The inputs are past usage history, preference information, and food expiration date data. Based on these, the server outputs a recipe optimized for the user. This recipe includes recipes that prioritize the use of ingredients with an approaching expiration date.

[0396] Step 6: Delivery of missing ingredients and equipment

[0397] The server checks whether the user has all the ingredients and equipment required for the generated recipe. If there are any shortages, it automatically generates a delivery request and sends an order to an external supply service. The input is the recipe data and the current list of ingredients and equipment, and the shortages are clearly displayed as output. Based on this shortage information, an order is sent to the delivery service, and the necessary ingredients and equipment are delivered to the user.

[0398] Step 7: Displaying results and cooking assistance

[0399] The server sends the final recipe data, cooking steps, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner, providing text and visual content to guide the user. For example, it displays detailed steps, cooking time, and a list of necessary equipment for making "Oyakodon." It also provides relaxing music and video links depending on the user's emotional state. The input data is the recipe details and guide information, and by displaying and playing this information, the device allows the user to smoothly proceed with cooking.

[0400] (Application example 2)

[0401] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0402] Conventional recipe suggestion systems do not fully consider the use of food in the refrigerator or the user's emotional state, making it difficult to provide a wide variety of recipe suggestions or services that respond to the user's emotions.In addition, in physical stores, there is a lack of services that suggest real-time recipes based on the food items that the user plans to purchase, or that encourage the purchase of necessary food and equipment, making it difficult to improve user satisfaction.

[0403] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating recipes based on the acquired food list and preference information, means for identifying and recognizing foods in the refrigerator through image analysis, means for referencing past usage data and personalizing recipes based on the user's preferences, means for automatically generating and sending a delivery request when necessary foods and utensils are missing, means for recognizing the user's emotional state and adjusting the recipe based on the emotional state, means for photographing foods to be purchased in the store and suggesting recipes, means for suggesting purchases when necessary foods and utensils are missing, and means for providing the user with recipes and cooking instructions. This enables a wide variety of recipes to be suggested based on the user's emotional state and the foods to be purchased, providing a comfortable shopping and cooking experience in a physical store.

[0404] "Acquired food list and preference information" refers to information indicating the food items currently owned by the user and the user's food preferences.

[0405] "Image analysis" is the technique of analyzing digital images to identify specific objects and extract information.

[0406] "Food in refrigerator" means food currently in a user's refrigerated storage unit.

[0407] "Past Usage Data" means historical information and related data generated by a User's use of the System.

[0408] "User Preferences" is information that indicates a user's personal preferences regarding foods and recipes that they like.

[0409] "Personalizing recipes" means adjusting recipes based on the user's individual preferences and past usage data, and suggesting recipes in the most optimal form for each individual user.

[0410] A "delivery request" is order information for delivery of necessary food and equipment to a specified location.

[0411] "User's emotional state" refers to the user's mental and emotional state as determined by analyzing voice and facial expression data.

[0412] "Foods to be purchased" refers to foods that a user intends to purchase in a physical store.

[0413] "Real-time recipe suggestions" means instantly generating and suggesting recipes based on the current situation and user input.

[0414] "Cooking instructions" are the detailed instructions and steps required to prepare a particular food product.

[0415] The system for realizing this invention is mainly composed of a server, a terminal, and a user. Each component and its operation will be explained below.

[0416] 1. Server Functions

[0417] The server has the following functions:

[0418] A means for generating recipes based on the obtained food list and preference information:

[0419] The server generates appropriate recipes based on the food list and cooking preferences entered or photographed by the user, and uses a generative AI model to suggest recipes that best match the ingredients and preferences.

[0420] Image analysis to identify and identify food items in refrigerators:

[0421] The system receives image data of the refrigerator sent by the user and uses an image analysis algorithm (e.g., Google Cloud Vision API) to identify the type and amount of food.

[0422] Ways to personalize recipes based on user preferences using past usage data:

[0423] The server uses past usage history data to suggest recipes tailored to the user's individual preferences, prioritizing recipes that have been popular in the past and ingredients that are frequently used.

[0424] A way to automatically generate and send delivery requests when needed food and equipment are in short supply:

[0425] If the server is running low on ingredients or cooking equipment required for the generated recipe, it will automatically send an order in conjunction with a delivery service and arrange for prompt delivery.

[0426] A way to recognize the user's emotional state and adjust recipes based on that emotional state:

[0427] The server analyzes voice and facial expression data to recognize the user's emotional state. For example, if the user is feeling stressed, it will prioritize recipes that are easy to prepare and have a relaxing effect.

[0428] A way to take a photo of the food you plan to buy in the store and get recipe suggestions:

[0429] The server takes a photo of the food the user plans to buy in a physical store, analyzes the image, and suggests suitable recipes, supporting the user's shopping and increasing their desire to buy.

[0430] Providing suggested purchases of necessary food and equipment if they are in short supply:

[0431] If the user is running low on food or equipment that he or she plans to purchase, the server will indicate this and suggest the purchase.

[0432] 2. Device Features

[0433] The device (e.g., smartphone, tablet) has the following functions:

[0434] Getting user input:

[0435] It offers a variety of input methods, including taking pictures of the refrigerator, entering a food list into a text box, and collecting voice and facial expression data.

[0436] Result display and cooking assistance:

[0437] The system supports cooking by displaying recipe data and cooking instructions sent from the server in an easy-to-understand manner to the user, and also provides relaxing music and video links as needed.

[0438] 3. User Behavior

[0439] Users use the system to perform the following actions:

[0440] Take and upload a picture of the food in your refrigerator.

[0441] Enter your ingredient list and cooking preferences into the text boxes.

[0442] Provides voice and facial expression data.

[0443] Order the necessary ingredients and equipment based on the proposed recipe.

[0444] Examples of specific examples and prompts

[0445] For example, if a user takes a photo of "tomatoes, cheese, pasta" in the refrigerator and the foods they have purchased, and provides the audio data to the app:

[0446] The server will parse the message assuming the following prompt:

[0447] Example prompt sentence:

[0448] "Generate the best recipe from the ingredients you have based on the user's emotional state. The current ingredients are tomatoes, cheese, and pasta."

[0449] This allows for flexible and personalized recipe suggestions tailored to the user's needs, providing a seamless shopping and cooking experience even in physical stores.

[0450] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0451] Step 1:

[0452] The user uses a smartphone or tablet to take pictures of the food in the refrigerator or the food they plan to buy. The user also inputs the food list into a text box and provides voice and facial expression data. The input data consists of image data, text data, and voice and facial expression data.

[0453] Step 2:

[0454] The device sends the collected image data, text data, and voice and facial expression data to the server, where it is entered as user input data.

[0455] Step 3:

[0456] The server uses image analysis algorithms to identify and identify the food in the refrigerator or the food you plan to buy from the image data sent to it. The extracted food data is output. At this stage, specific food names such as "tomato," "cheese," and "pasta" are obtained from the image data.

[0457] Step 4:

[0458] The server analyzes the voice and facial expression data to recognize the user's emotional state. It uses an emotion recognition engine (e.g., Google Cloud Speech-to-Text) to perform the analysis and outputs the user's emotional state. For example, it determines whether the user is relaxed or stressed.

[0459] Step 5:

[0460] The server uses a generative AI model based on the acquired food list and preference information to generate recipes. The input is food data and emotional state data, and the output is a suggested recipe. At this stage, a specific recipe such as "pasta with tomato and cheese" is output.

[0461] Step 6:

[0462] The server references past usage data and personalizes recipes based on the user's preferences. Entering past usage data results in personalized recipes. For example, if a user has previously preferred "tomato pasta," the recipe will be adjusted to take this into account.

[0463] Step 7:

[0464] The server automatically generates and sends a delivery request if the food and equipment required for the generated recipe are missing. The input is the missing food and equipment data, and the output is the generated delivery request. For example, if "cheese" is missing, a request is generated suggesting the purchase of that item.

[0465] Step 8:

[0466] The server sends the final recipe data, cooking instructions, and information on missing foods and equipment to the device. The input is the generated recipe and cooking instructions data, and the output is the information sent to the device. Specifically, the step-by-step cooking instructions and instructions for purchasing "cheese" are displayed.

[0467] Step 9:

[0468] The device finally displays the recipe and cooking instructions to the user, and provides relaxing music and video links as needed.The user can then cook based on the displayed information and enjoy the suggested recipe.

[0469] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0470] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0471] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0472] [Second embodiment]

[0473] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0474] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0475] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0476] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0477] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0478] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0479] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0480] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0481] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0482] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0483] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0484] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0485] The present invention provides a system for solving the problems of cooking in a rut and food waste in busy daily lives. Below, an embodiment of this system is described.

[0486] 1. Getting User Input

[0487] To help users efficiently utilize the ingredients in their refrigerator and enjoy a wide variety of dishes, the device first acquires input data. The device provides users with the options to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their favorite dishes. This allows the device to collect information about the user.

[0488] 2. Image Analysis and Data Conversion

[0489] The image data of the refrigerator sent by the device is received by the server. The server uses an image analysis algorithm to identify and identify the ingredients present in the refrigerator. At the same time, the list of ingredients and cooking preferences are also analyzed and converted into text data. For example, ingredients such as "carrots," "bell peppers," "chicken," and "eggs" can be recognized from the image and recorded as text data.

[0490] 3. Recipe Creation and Search

[0491] The server generates recipes based on the acquired ingredient list and preference information. Using a generative AI model, it suggests recipes that make the most of the ingredients the user has. For example, if a user has chicken, eggs, carrots, and green peppers, it will suggest recipes such as oyakodon (chicken and egg rice bowl) or stir-fry. It also references an existing recipe database to search for and provide related recipe information.

[0492] 4. Personalizing your data

[0493] The server refers to the user's past usage data and personalizes recipes based on the user's preferences and habits. It also takes into account the expiration dates of ingredients and makes suggestions to minimize food waste. For example, it can prevent food waste by suggesting recipes that prioritize ingredients with a short expiration date.

[0494] 5. Delivery of missing ingredients and equipment

[0495] If the server does not have the ingredients or equipment required for the generated recipe on hand, it automatically generates a delivery request and sends an order to an external supply service. The delivery service quickly arranges for the missing ingredients and equipment and delivers them to the user. For example, if there is a shortage of "chicken," that information is sent to the delivery service and arrangements are made to procure it.

[0496] 6. Results display and cooking assistance

[0497] The server sends the final recipe data, cooking instructions, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner to the user, providing text and visual content to guide the user through the cooking methods and procedures. For example, the device displays detailed instructions, cooking time, and a list of necessary equipment for making "Oyakodon."

[0498] Specific examples

[0499] Example 1: Upload an image of a refrigerator

[0500] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app.

[0501] 2. The server analyzes the image and recognizes "carrots," "bell peppers," "chicken," and "eggs."

[0502] 3. The server generates a recipe for "Oyakodon" based on these ingredients and sends the cooking instructions to the terminal.

[0503] 4. The device displays cooking instructions to the user.

[0504] Example 2: Enter a list of ingredients

[0505] 1. A user types "tomato, cheese, pasta" into a text box.

[0506] 2. The server analyzes this, searches for related recipes, and suggests "pasta with tomatoes and cheese."

[0507] 3. The server generates a recipe that takes expiration dates into account and sends the details to the device.

[0508] 4. The device displays the recipe and cooking instructions to the user.

[0509] In this way, the present invention can effectively utilize leftover ingredients in the refrigerator, preventing daily cooking from becoming monotonous and providing users with a fresh and healthy diet.

[0510] The processing flow will be explained below.

[0511] Step 1:

[0512] A user launches the application using a device such as a smartphone or tablet, which then provides the user with the option to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their preferred dish.

[0513] Step 2:

[0514] Users can take a picture of their refrigerator and upload it to the application, or enter their ingredient list and cooking preferences into a text box.

[0515] Step 3:

[0516] The terminal sends the user's input data (image data or text data) to the server.

[0517] Step 4:

[0518] The server uses an image analysis algorithm to analyze the image data received from the refrigerator. Specifically, it applies a food recognition model to identify ingredients from the image. This allows ingredients such as "carrots," "bell peppers," "chicken," and "eggs" to be identified and classified.

[0519] Step 5:

[0520] The server performs text analysis on the user's input of ingredients and cooking preferences to extract ingredients and cooking genres, which are also used to generate recipes.

[0521] Step 6:

[0522] The server uses the generative AI model to generate customized recipes based on the user's food list and preferences, and also references existing recipe databases such as Classy to search and retrieve related recipe information.

[0523] Step 7:

[0524] The server references the user's past usage data, taking into account information such as past favorite recipes and commonly used ingredients, and personalizes recipes based on the user's preferences.

[0525] Step 8:

[0526] The server creates a list of ingredients and equipment needed for the generated recipe, checking the user's ingredient list to ensure all ingredients are available.

[0527] Step 9:

[0528] The server adjusts recipes taking into account the expiration dates of ingredients, suggesting recipes that prioritize ingredients with close expiration dates and preventing food waste.

[0529] Step 10:

[0530] If the server is running low on ingredients or equipment, it automatically generates a delivery request, which is then sent to an external supply service (e.g., an online shopping site).

[0531] Step 11:

[0532] The server receives confirmation from the delivery service and sends a notification to the user, for example, "Your chicken will arrive in 30 minutes."

[0533] Step 12:

[0534] The server sends the final recipe data, cooking instructions, and information on missing ingredients and utensils to the terminal.

[0535] Step 13:

[0536] The device receives information and displays it to the user in an easy-to-understand manner, such as providing recipe steps, necessary equipment, and links to cooking videos.

[0537] Step 14:

[0538] The user begins cooking according to the recipe displayed on the device, with the help of a timer and additional tips provided by the app.

[0539] Step 15:

[0540] The device records the user's cooking progress, sends the data back to the server, and stores it in a database for future personalization.

[0541] Example 1

[0542] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0543] In today's busy lifestyles, cooking at home can easily become monotonous, and food in the refrigerator is often underutilized and thrown away. While this problem differs from household to household, the following issues need to be resolved: First, to efficiently utilize the food in the refrigerator and provide nutritionally balanced meals. Second, to suggest personalized recipes that take into account the user's preferences and past cooking history. Third, to assist the user in cooking by quickly arranging for ingredients and utensils that are in short supply.

[0544] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0545] In this invention, the server includes means for generating recipes based on the acquired ingredient list and preference information, means for identifying and identifying ingredients in the refrigerator through image analysis, means for referencing past usage data and personalizing recipes based on the user's preferences, means for automatically generating and sending a delivery request when necessary ingredients and cooking utensils are in short supply, means for providing the user with recipes and cooking instructions, means for generating recipes based on input data using a generative AI model, and means for automatically generating prompt statements, inputting them into the AI ​​model, and generating recipes based on the results. This makes it possible to efficiently use ingredients in the refrigerator, provide personalized recipes that suit the user's preferences, and minimize food waste.

[0546] The "obtained ingredient list" refers to the information about ingredients in the refrigerator entered by the user, including image analysis results and text input data.

[0547] "Preference information" is data about a user's preferences based on their cooking preferences, past usage history, etc.

[0548] The "means for generating recipes" is a function that creates new recipes using a generative AI model and recipe database based on the acquired ingredient list and preference information.

[0549] "Image analysis" is a technique that processes image data to identify ingredients in the refrigerator. It includes image recognition algorithms and machine learning models.

[0550] "Means for identification" refers to the function of identifying and specifying ingredients based on certain criteria from the information obtained by image analysis.

[0551] "Past Usage Data" refers to all data from a user's previous use of the system, including information about recipes selected and ingredients used.

[0552] "Personalization" refers to a function that suggests individually appropriate recipes based on the user's past usage data and preferences.

[0553] "Delivery Request" refers to a request to send an order to an external supply service when necessary ingredients or cooking equipment are in short supply.

[0554] "Means for providing cooking instructions" is a function that clearly provides specific steps and information on necessary equipment when a user cooks a dish.

[0555] A "generative AI model" is an artificial intelligence model used for complex data processing such as image analysis and recipe generation, as well as natural language processing.

[0556] A "prompt sentence" is a text sentence that expresses a question or request that is input to a generative AI model, and the AI ​​generates a response based on this.

[0557] The present invention is a system that solves the problems of cooking in a rut and food waste in busy daily lives. This system efficiently utilizes ingredients in the refrigerator and suggests a wide variety of dishes, providing users with a fresh and healthy diet. Detailed embodiments for implementing this system are described below.

[0558] Getting User Input

[0559] Users use a dedicated application on their smartphone or PC to input information about the ingredients in their refrigerator. The input method is as follows:

[0560] 1. Take a picture of the ingredients in your refrigerator and upload it.

[0561] 2. Enter the ingredients list in the text box.

[0562] 3. Use the drop-down menu to select your preferred cuisine.

[0563] Image analysis and data conversion

[0564] The device sends image data, text data, or data on the user's favorite dishes to the server. The server then uses an image analysis algorithm (using, for example, OpenCV or TensorFlow) to identify the ingredients in the refrigerator. For example, it identifies "carrots," "bell peppers," "chicken," and "eggs" from the image and saves them as text data. This allows the server to obtain a list of ingredients and the user's favorite dishes.

[0565] Recipe Generation

[0566] The server generates a recipe using a generative AI model (for example, GPT-3 or a similar natural language processing model) based on the acquired ingredient list and preference information. Specifically, the server automatically generates a prompt sentence and inputs it into the generative AI model. For example, a prompt sentence such as "What recipes use chicken, eggs, carrots, and green peppers?" is generated and input into the AI ​​model. The recipe is generated based on the results obtained from the AI ​​model.

[0567] Data Personalization

[0568] The server refers to the user's past usage data and proposes personalized recipes that take into account the user's preferences and the expiration dates of ingredients, thereby prioritizing the use of ingredients with an approaching expiration date, thereby minimizing food waste.

[0569] Delivery of missing ingredients and cooking equipment

[0570] If the server is running low on ingredients or cooking utensils required for the generated recipe, it automatically generates a delivery request and sends it to an external supply service. For example, if there is a shortage of chicken, the server sends that information to the delivery service and quickly arranges for the missing ingredients or cooking utensils.

[0571] Result display and cooking assistance

[0572] The server sends the final recipe data, cooking instructions, and information on missing ingredients and cooking utensils to the device. The device then displays this information in an easy-to-understand manner to the user, guiding them through the cooking methods and procedures. For example, the device displays detailed instructions for making an "oyakodon," cooking time, and a list of necessary utensils. It also displays text and visual content (images and videos) to guide the user through the cooking process.

[0573] Specific examples

[0574] Example 1: Upload an image of a refrigerator

[0575] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app.

[0576] 2. The server analyzes the image and identifies "carrots," "bell peppers," "chicken," and "eggs."

[0577] 3. The server generates a recipe for "Oyakodon" based on these ingredients and sends the cooking instructions to the terminal.

[0578] 4. The device displays cooking instructions to the user.

[0579] Example 2: Enter a list of ingredients

[0580] 1. A user types "tomato, cheese, pasta" into a text box.

[0581] 2. The server analyzes this, searches for related recipes, and suggests "pasta with tomatoes and cheese."

[0582] 3. The server generates a recipe that takes expiration dates into account and sends the details to the device.

[0583] 4. The device displays the recipe and cooking instructions to the user.

[0584] In this way, users can efficiently utilize the ingredients in their refrigerator and enjoy fresh and healthy meals without getting bored with their daily cooking routine.

[0585] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0586] Step 1: Getting User Input

[0587] Users launch the application on their smartphone or PC and enter information about the ingredients in their refrigerator by taking a picture of the refrigerator and uploading it, entering the list of ingredients in a text box, or selecting the option to choose their favorite dish.

[0588] Input: Image data of the contents of the refrigerator, text data, or favorite food information

[0589] Output: User input data (image data, text data, favorite food information)

[0590] Step 2: Image analysis and data conversion

[0591] The terminal transmits the acquired user input data to the server.

[0592] The server processes the received image data using an image analysis algorithm (e.g., OpenCV or TensorFlow) to identify the ingredients in the refrigerator. For example, it identifies "carrots," "bell peppers," "chicken," and "eggs" from the image.

[0593] Input: User-entered data (image data, text data, favorite food information)

[0594] Data processing: Identification and classification of ingredients using image analysis

[0595] Output: Identified ingredients list (text data)

[0596] Step 3: Generate the recipe

[0597] Based on the ingredient list and preference information acquired by the server, a recipe is generated using a generative AI model (e.g., GPT-3). The server generates a prompt sentence and inputs it into the generative AI model. For example, a prompt sentence such as "What is a recipe that uses chicken, eggs, carrots, and green peppers?" is generated and input into the AI ​​model.

[0598] Input: Ingredient list, preference information

[0599] Data processing: Recipe generation using generative AI models

[0600] Output: Suggested recipe (text data)

[0601] Step 4: Personalize your data

[0602] The server references the user's past usage data and generates personalized recipes that take into account the user's preferences and the expiration dates of ingredients. Specifically, it suggests recipes that prioritize ingredients with an approaching expiration date.

[0603] Input: Ingredient list, preference information, past usage data

[0604] Data processing: Consideration of preference analysis and expiration date data

[0605] Output: Personalized recipe (text data)

[0606] Step 5: Delivery of missing ingredients and cooking equipment

[0607] The server creates a list of ingredients and cooking tools needed based on the recipe. If any of the ingredients are missing, it automatically generates a delivery request and sends it to an external supply service. For example, if chicken is in short supply, it requests a delivery service to procure it.

[0608] Input: Personalized recipe, information about ingredients and cooking equipment the user has on hand

[0609] Data processing: Identifying missing ingredients and cooking utensils, generating automatic delivery requests

[0610] Output: Delivery request

[0611] Step 6: Displaying results and cooking assistance

[0612] The server sends the final recipe data, along with information on missing ingredients and cooking utensils, to the device. The device then displays this information in an easy-to-understand manner to the user, guiding them through the cooking method and steps. For example, it displays detailed instructions, cooking time, and a list of necessary utensils for making "Oyakodon."

[0613] Input: Final recipe data, cooking instructions, missing ingredients and utensils information

[0614] Data processing: Format conversion of cooking instructions and necessary information

[0615] Output: Cooking guide display for the user (text and visual content)

[0616] Through these steps, the system can efficiently utilize the ingredients in the refrigerator and provide users with a fresh and healthy diet.

[0617] (Application example 1)

[0618] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0619] Conventional food management systems do not adequately suggest recipes that efficiently utilize ingredients in the refrigerator, resulting in food waste. Furthermore, recipe suggestions that take into account the user's preferences and past cooking history are limited. In particular, the lack of a convenient way to provide cooking instructions or delivery of missing ingredients in busy daily lives has been a major problem for users.

[0620] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0621] In this invention, the server includes: means for generating recipes based on the acquired food list and preference information; means for identifying and recognizing foods in the refrigerator through image analysis; means for referencing past usage data and personalizing recipes based on the user's preferences; means for automatically generating and sending delivery requests when necessary foods and utensils are missing; means for providing recipes and cooking instructions to the user; means for processing food images taken with a smartphone; means for automatically recognizing the acquired food list using an image analysis algorithm; means for using a generative AI model to generate and suggest optimal recipes based on the acquired food list; and means for transmitting the missing list to a delivery service. This enables effective use of ingredients in the refrigerator, reduction of food waste, personalized recipe suggestions for users, and easy delivery ordering.

[0622] - "Food List" refers to a list of ingredients owned by the user.

[0623] "Preference information" refers to information about the cuisine and taste preferences of a user that has been selected in the past.

[0624] The "means for generating recipes" is a function that automatically creates recipes showing cooking methods and ingredients based on the acquired food list and preference information.

[0625] "Image analysis" refers to techniques and processes that process digital image data to extract specific information.

[0626] "Means for identifying and discerning food in a refrigerator" refers to a technology that uses image analysis to recognize and identify food items present in a refrigerator.

[0627] "Past usage data" refers to historical information such as ingredients used by the user, recipes selected, and ingredients ordered.

[0628] "Means for personalizing recipes based on user preferences" refers to technology that references past usage data and suggests recipes tailored to the user's specific preferences.

[0629] "Means for automatically generating and sending delivery requests" refers to a function that automatically detects when necessary food or equipment is missing and sends an order to an external delivery service.

[0630] "Means for providing recipes and cooking instructions" refers to a function that displays or communicates suggested recipes and their cooking instructions to the user.

[0631] The "means for processing food images taken by a smartphone" refers to the process of receiving image data taken by a user, analyzing it, and extracting a list of foods to be used.

[0632] An "image analysis algorithm" is a set of mathematical models or programs that process image data to recognize specific objects or text information.

[0633] A "generative AI model" is an artificial intelligence model that uses machine learning to generate new recipes and information from data.

[0634] A "delivery service" is a service organization that delivers ingredients and equipment to users.

[0635] The present invention is a system that efficiently utilizes ingredients in a refrigerator, provides a wide variety of dishes, and prevents food waste. This system starts when a user takes a photo of the ingredients in the refrigerator with their smartphone and uploads it to an application.

[0636] Hardware and software used

[0637] Hardware:

[0638] Smartphone: Taking pictures of ingredients and using applications.

[0639] Server: Image analysis, recipe generation, and delivery request processing.

[0640] Delivery service computer system: Arranges delivery of ingredients and cooking equipment.

[0641] software:

[0642] Flask: A web application framework in Python.

[0643] PIL (Pillow): Image processing library.

[0644] food_recognition: Food recognition algorithm.

[0645] recipe_generation: Recipe generation algorithm.

[0646] delivery_service: Third-party delivery service API.

[0647] Processing flow

[0648] 1. Getting User Input

[0649] Users take a photo of the ingredients in their refrigerator with their smartphone and upload the image to the application.

[0650] Alternatively, you're given the option to manually enter the ingredient list into a text box.

[0651] Information about the user's preferences and past cooking tastes can also be entered.

[0652] 2. Image Analysis and Data Conversion

[0653] The server receives the image data sent from the smartphone and reads the image data using Pillow.

[0654] Ingredients are identified using an image analysis algorithm (food_recognition), and the information is converted into text data.

[0655] For example, "tomato," "cheese," and "beef" can be identified from an image of a refrigerator and recorded as text data.

[0656] 3. Recipe Generation

[0657] A generative AI model (recipe_generation) is used to generate recipes based on the acquired ingredient list and user preference information.

[0658] For example, we suggest a recipe for Bolognese pasta using tomatoes, cheese, and beef.

[0659] 4. Creating a delivery request

[0660] The server checks the ingredients and equipment required for the generated recipe, and if any are missing, it automatically generates a delivery request for them.

[0661] It calls the delivery service's API and arranges for the necessary ingredients and equipment to be delivered to the user.

[0662] 5. Results display and cooking assistance

[0663] The server sends the final recipe data, cooking instructions, and information on missing ingredients and utensils to the smartphone.

[0664] The smartphone displays this information in an easy-to-understand way for the user, guiding them through the cooking process with visual and textual content.

[0665] Specific examples

[0666] Example 1: Uploading food images

[0667] Users take a picture of the inside of their refrigerator with their smartphone and upload it to the app.

[0668] The server analyzes the image and recognizes "carrots," "bell peppers," "chicken," and "eggs."

[0669] The server uses a generative AI model to generate a recipe for "Oyakodon" based on these ingredients and sends the cooking instructions to a smartphone.

[0670] The smartphone displays cooking instructions to the user.

[0671] Example 2: Ordering delivery of ingredients that are in short supply

[0672] A user uploads a picture of their refrigerator and discovers that they are out of "chicken."

[0673] The server generates a delivery request and places an order for "chicken" through the delivery service API.

[0674] The delivery service arranges for the chicken to be delivered to the user.

[0675] Example 3: Example of a prompt statement

[0676] Prompt: "How can I build a system that allows you to upload a picture of the ingredients in your refrigerator, suggest recipes based on the images, and order delivery for any missing ingredients? What are the basic features this system should have?"

[0677] As a result, this system can make effective use of ingredients in the refrigerator, reduce food waste, and provide users with personalized recipe suggestions and delivery services.

[0678] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0679] Step 1:

[0680] The user takes a photo of the food in the refrigerator with their smartphone and uploads the image to the application.

[0681] Specific actions

[0682] Users use their smartphone camera to take a picture of the ingredients in their refrigerator and upload the image to a dedicated app, which also offers the option to input user preference information and allows users to provide past usage data.

[0683] Input and Output

[0684] Input: Image data of ingredients, preference information

[0685] Output: Image data and preference information sent to the server

[0686] Step 2:

[0687] The server receives the image data and loads the image using Pillow.

[0688] Specific actions

[0689] The server receives the uploaded image data and reads the image using the Pillow image processing library, thereby obtaining the digital image data.

[0690] Input and Output

[0691] Input: Image data sent from a smartphone

[0692] Output: Loaded image data

[0693] Step 3:

[0694] The server uses an image analysis algorithm (food_recognition) to identify ingredients and convert them into text data.

[0695] Specific actions

[0696] The server analyzes the received image data using the food_recognition algorithm, identifies the ingredients in the image, converts the identified ingredients into text data, and creates a list.

[0697] Input and Output

[0698] Input: Image data

[0699] Output: Ingredient list (text data)

[0700] Step 4:

[0701] The server uses a generative AI model (recipe_generation) to generate a recipe based on the ingredient list and preference information.

[0702] Specific actions

[0703] The server uses a generative AI model to create an appropriate recipe based on the acquired ingredient list and the user's preferences. The generated recipe also includes specific cooking steps.

[0704] Input and Output

[0705] Input: Ingredient list, preference information

[0706] Output: Generated recipe and cooking instructions

[0707] Step 5:

[0708] The server checks for missing ingredients and equipment based on the generated recipe and generates a delivery request.

[0709] Specific actions

[0710] The server compares all ingredients included in the generated recipe with the user's current inventory, identifies any missing ingredients or utensils, and generates and sends a request to an external delivery service API.

[0711] Input and Output

[0712] Input: Generated recipe, ingredient list

[0713] Output: Delivery request

[0714] Step 6:

[0715] The server sends the final recipe data, cooking instructions, and delivery request results to the smartphone.

[0716] Specific actions

[0717] The server then sends the generated recipe data, cooking instructions, and delivery results for missing ingredients and utensils to a smartphone, where users can check the information using the smartphone app.

[0718] Input and Output

[0719] Input: Final recipe data, cooking instructions, delivery request results

[0720] Output: Recipe data and cooking instructions displayed on a smartphone

[0721] Through these steps, the system can effectively utilize the ingredients in the refrigerator and provide personalized recipe suggestions and delivery services.

[0722] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0723] The present invention is a system that efficiently utilizes ingredients and utensils that a user has, and also suggests recipes and cooking methods according to the user's emotional state. The following describes an embodiment of this system.

[0724] 1. Getting User Input

[0725] To help users efficiently utilize the ingredients in their refrigerator and enjoy a wide variety of dishes, the device first acquires input data. The device provides users with the option to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their preferred dishes. Additionally, the device also has the ability to collect voice and facial expression data to recognize the user's emotions.

[0726] 2. Image Analysis and Data Conversion

[0727] The image data of the refrigerator sent by the device is received by the server. The server uses an image analysis algorithm to identify and identify the ingredients present in the refrigerator. At the same time, the list of ingredients and cooking preferences are also analyzed and converted into text data. For example, ingredients such as "carrots," "bell peppers," "chicken," and "eggs" can be recognized from the image and recorded as text data.

[0728] 3. Emotional Recognition

[0729] The device collects the user's voice and facial expression data and sends it to the server. The server then uses an emotion engine to analyze and recognize the user's emotions from this data. For example, it analyzes changes in voice tone and facial expressions to determine whether the user is relaxed or stressed.

[0730] 4. Recipe Creation and Search

[0731] The server generates recipes based on the acquired ingredient list and preference information. Using a generative AI model, it suggests recipes that make the most of the ingredients the user has. It also references an existing recipe database to search for and provide related recipe information. For example, if a user has chicken, eggs, carrots, and green peppers, it will suggest recipes such as oyakodon (chicken and egg rice bowl) or stir-fry. Depending on the user's emotional state, it also prioritizes suggestions of very simple recipes and dishes with a relaxing effect.

[0732] 5. Personalization of Data

[0733] The server refers to the user's past usage data and personalizes recipes based on the user's preferences and habits. It also takes into account the expiration dates of ingredients and makes suggestions to minimize food waste. For example, it can prevent food waste by suggesting recipes that prioritize ingredients with a short expiration date.

[0734] 6. Delivery of missing ingredients and equipment

[0735] If the server does not have the ingredients or equipment required for the generated recipe on hand, it automatically generates a delivery request and sends an order to an external supply service. The delivery service quickly arranges for the missing ingredients and equipment and delivers them to the user. For example, if there is a shortage of "chicken," that information is sent to the delivery service and arrangements are made to procure it.

[0736] 7. Results display and cooking assistance

[0737] The server sends the final recipe data, cooking instructions, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner to the user, providing text and visual content to guide the user through the cooking methods and procedures. For example, the device may display detailed instructions, cooking time, and a list of necessary equipment for making "Oyakodon." Furthermore, the device may also provide relaxing music and video links depending on the user's emotional state.

[0738] Specific examples

[0739] Example 1: Uploading images and audio data of a refrigerator

[0740] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app, which also collects audio data.

[0741] 2. The server analyzes the image and recognizes "carrot," "bell pepper," "chicken," and "egg." At the same time, it analyzes the voice data and recognizes the user's emotional state.

[0742] 3. The server generates a recipe for "Oyakodon" based on this information and sends the cooking instructions to the device. If the user is feeling stressed, it also suggests relaxing music.

[0743] 4. The device displays and plays cooking instructions and relaxing music to the user.

[0744] Example 2: Input ingredients list and facial expression data

[0745] 1. The user enters "tomato, cheese, pasta" into the text box, and facial expression data is collected using the camera.

[0746] 2. The server analyzes this, searches for related recipes, and suggests "Tomato and Cheese Pasta." It also determines from facial expression data that the user is relaxed.

[0747] 3. The server generates a recipe that takes expiration dates into account and sends the details to the device.

[0748] 4. The device displays the recipe and cooking instructions to the user.

[0749] In this way, the present invention effectively utilizes leftover ingredients in the refrigerator and suggests recipes that take into consideration the user's emotional state, preventing cooking from becoming monotonous, reducing food waste, and providing a richer diet.

[0750] The processing flow will be explained below.

[0751] Step 1:

[0752] A user launches the application using a device such as a smartphone or tablet. The device provides the user with the options to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their preferred dish. It also prompts them to collect voice data and capture facial expression data.

[0753] Step 2:

[0754] Users take a picture of their refrigerator and upload it to the application, and are also asked to record a voice message or enter facial expression data via the camera to ascertain its emotional state.

[0755] Step 3:

[0756] The terminal sends the user's input data (image data, voice data, facial expression data, or text data) to the server.

[0757] Step 4:

[0758] The server analyzes the image data of the refrigerator it receives using an image analysis algorithm. Specifically, it uses a food recognition model to identify ingredients from the image. As a result, ingredients such as "carrots," "bell peppers," "chicken," and "eggs" in the refrigerator are identified.

[0759] Step 5:

[0760] The server analyzes the user's ingredient list and cooking preferences received as text data and extracts the ingredient list and cooking categories.

[0761] Step 6:

[0762] The server uses an emotion engine to analyze the transmitted voice and facial expression data and recognize the user's emotional state. For example, it can determine whether the user is relaxed or stressed based on changes in voice tone and facial expression.

[0763] Step 7:

[0764] The server generates recipes based on the acquired ingredient list and the user's recognized emotional state. Using a generative AI model, the server suggests recipes that make the most of the ingredients the user has. The server then adjusts the recipes according to the user's emotional state. For example, if the user is tired, the server suggests simple recipes that have a relaxing effect.

[0765] Step 8:

[0766] The server uses past usage data to personalize recipes based on the user's preferences and habits, and also takes into account expiration dates of ingredients, prioritizing recipes that minimize food waste.

[0767] Step 9:

[0768] The server creates a list of ingredients and tools required for the generated recipe, referencing the user's ingredient list to see what is available and what is missing.

[0769] Step 10:

[0770] If the server is missing ingredients or equipment, it automatically generates a delivery request, which is then sent to an external supply service.

[0771] Step 11:

[0772] The server receives confirmation from the delivery service and notifies the user when the missing ingredients or equipment will arrive, for example, "The chicken will arrive in 30 minutes."

[0773] Step 12:

[0774] The server sends the final recipe data, cooking instructions, and information on missing ingredients and utensils to the terminal.

[0775] Step 13:

[0776] The device displays the information received in an easy-to-understand manner to the user, providing recipe steps, necessary utensils, and links to cooking videos through text and visual content. It also displays relaxing music and videos based on the user's emotional state.

[0777] Step 14:

[0778] The user begins cooking by following the recipe displayed on the device, with the help of a timer and additional tips provided by the app.

[0779] Step 15:

[0780] The device records the user's cooking progress, sends the data back to the server, and stores it in a database for future personalization.

[0781] Example 2

[0782] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0783] Currently, many households are unable to efficiently utilize the ingredients in their refrigerators, resulting in food waste. It is also difficult for systems to suggest dishes that reflect the user's emotional state, which can reduce meal satisfaction. Furthermore, if the necessary ingredients or cooking utensils are in short supply, it takes time to procure them, making it difficult to start cooking quickly. There is a need to solve these issues and provide users with an efficient and satisfying cooking experience.

[0784] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0785] In this invention, the server includes means for receiving and analyzing user input as text data and image data, means for identifying and identifying foods in the refrigerator through image analysis, means for generating recipes based on the acquired food list and preference information, means for analyzing the user's emotional state using voice and facial expression data, means for suggesting recipes according to the user's emotional state, means for personalizing recipes based on the user's preferences by referencing past usage data, means for automatically generating and sending a delivery request when necessary foods and utensils are in short supply, means for providing the user with recipes and cooking instructions, and means for generating recipes using a generative AI model. This makes it possible to efficiently use ingredients in the refrigerator, suggest recipes according to the user's emotional state, and quickly procure the necessary ingredients and utensils.

[0786] "User input" refers to text data, image data, voice data, and facial expression data that a user provides to the system.

[0787] "Text data" refers to data entered by the user in the form of text, such as a list of ingredients or preference information.

[0788] "Image data" refers to images of food and other visual information in the refrigerator taken with a camera and uploaded to the system.

[0789] "Audio Data" refers to audio information collected for the purpose of analyzing a user's emotional state.

[0790] "Facial expression data" refers to emotional information extracted from images of a user's facial expressions captured by a camera.

[0791] "Image analysis" refers to the process of analyzing transmitted image data to identify and identify the food items in the refrigerator.

[0792] A "food list" refers to data that records in text format all the food in your refrigerator.

[0793] "Preference information" refers to information that represents a user's preferences for food and cooking.

[0794] "Emotional state" refers to the user's current psychological and emotional state, analyzed from their voice and facial expression data.

[0795] A "recipe" is a document or piece of information that lists the ingredients and steps needed to prepare a particular dish.

[0796] "Generative AI model" refers to an algorithmic model that uses artificial intelligence to generate new recipes.

[0797] "Past usage data" refers to data that records a user's system usage history.

[0798] "Food waste" refers to food that is discarded without being consumed.

[0799] "Delivery Request" means an order request sent to an external supply service to procure needed food and / or equipment.

[0800] "Personalization" refers to the process of providing users with optimized suggestions and services based on their past usage history and preferences.

[0801] The present invention is a system that efficiently utilizes the ingredients and cooking utensils that a user has, and also suggests recipes and cooking methods that correspond to the user's emotional state. Specific implementation methods of this system are described in detail below.

[0802] Getting User Input

[0803] Users input ingredient information and their favorite dishes using a smartphone or PC. The device provides users with the ability to take and upload images of their refrigerator, a text box for entering a list of ingredients, and check boxes and drop-down menus for selecting their favorite dishes. The device also collects user emotional data through voice input and facial recognition using a camera.

[0804] Image analysis and data conversion

[0805] The server receives the image data of the refrigerator sent by the device and uses image analysis algorithms (e.g., OpenCV or TensorFlow) to identify and distinguish the ingredients in the refrigerator. For example, it recognizes "carrots," "bell peppers," "chicken," and "eggs" from the image of the refrigerator and records them as text data. In addition, it analyzes the text list of ingredients and cooking preferences entered by the user and stores them in a database.

[0806] Emotion recognition

[0807] The device sends the collected voice and facial expression data to a server, which then analyzes it using an emotion analysis engine (e.g., IBM Watson or Microsoft Azure's emotion analysis API). This determines whether the user is relaxed or stressed. The analysis results are then recorded in a database.

[0808] Recipe creation and search

[0809] The server uses a generative AI model (e.g., GPT-4) to generate new recipes based on the acquired ingredient list and emotional data. For example, if a user has chicken, eggs, carrots, and green peppers, it will suggest recipes such as oyakodon (chicken and egg rice bowl) or stir-fry. It will also search an existing recipe database to provide related recipe information. If the user is feeling stressed, it will prioritize suggestions of easy recipes and dishes that have a relaxing effect.

[0810] Data Personalization

[0811] The server refers to the user's past usage data and personalizes recipes based on the user's preferences. For example, if there is an ingredient that is close to its expiration date, it can suggest a recipe that prioritizes using that ingredient, thereby reducing food waste.

[0812] Delivery of missing ingredients and equipment

[0813] The server checks whether the user has the ingredients and equipment required for the generated recipe. If they are in short supply, it automatically generates a delivery request and sends an order to an external supply service. For example, if there is a shortage of "chicken," that information is sent to the delivery service and arrangements are made for timely procurement.

[0814] Result display and cooking assistance

[0815] The server sends the final recipe data, cooking instructions, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner to the user, providing text and visual content to guide the user through the cooking method and steps. For example, it may display detailed instructions, cooking time, and a list of necessary equipment for making "Oyakodon." It may also provide relaxing music or video links depending on the user's emotional state.

[0816] Specific examples

[0817] Example 1: Uploading images and audio data of a refrigerator

[0818] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app, which also collects audio data.

[0819] 2. The device sends image and audio data to the server.

[0820] 3. The server analyzes the image and recognizes "carrot," "bell pepper," "chicken," and "egg." It also analyzes the voice data to understand the user's emotional state.

[0821] 4. The server generates a recipe for "Oyakodon" based on the recognized ingredients and emotional data. If it determines that the user is feeling stressed, it selects music with a relaxing effect.

[0822] 5. The server sends the generated recipe and music information to the device.

[0823] 6. The device displays and plays cooking instructions and relaxing music to the user.

[0824] Example 2: Input ingredients list and facial expression data

[0825] 1. The user enters "tomato, cheese, pasta" into the text box, and facial expression data is collected using the camera.

[0826] 2. The device sends the text and facial expression data to the server.

[0827] 3. The server analyzes the text data to search for relevant recipes and determines whether the user is relaxed based on facial expression data.

[0828] 4. The server generates a recipe for "Tomato and Cheese Pasta" and creates a recipe that takes into account the expiration date.

[0829] 5. The server sends the generated recipe to the device.

[0830] 6. The device displays the recipe and cooking instructions to the user.

[0831] In this way, the present invention effectively utilizes leftover ingredients in the refrigerator and suggests recipes that take into consideration the user's emotional state, preventing cooking from becoming monotonous, reducing food waste, and providing a richer diet.

[0832] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0833] Step 1: Getting User Input

[0834] Users open the app on their smartphone or PC and enter information about ingredients in their refrigerator and their favorite dishes, or take and upload images. Voice input and facial expression data from the camera are also collected. Image data, text data, voice data, and facial expression data are obtained as input, and this input data is sent to the system. The device collects this data, converts it into an appropriate format, and sends it to the server.

[0835] Step 2: Image analysis and data conversion

[0836] The server receives the image data of the refrigerator sent from the device and uses image analysis algorithms (e.g., OpenCV or TensorFlow) to identify the ingredients in the refrigerator. In this process, image data is input and converted into text data. For example, it identifies "carrots," "bell peppers," "chicken," and "eggs" from the image of the refrigerator and outputs them as text data. The server also analyzes the text list of ingredients and cooking preferences entered by the user and stores them in a database.

[0837] Step 3: Recognize emotions

[0838] The server receives the voice and facial expression data sent from the device and analyzes the user's emotional state using an emotion analysis engine (e.g., IBM Watson or Microsoft Azure's emotion analysis API). Voice and facial expression data are received as input, and they are analyzed to output information about the user's emotional state (relaxed, stressed, etc.). The analysis results are stored in a database.

[0839] Step 4: Creating and searching recipes

[0840] The server generates new recipes using a generative AI model (e.g., GPT-4) based on the acquired ingredient list, emotional data, and preference information. In this process, the ingredient list and emotional state are input as text data, and recipe suggestions based on them are output. For example, if a user has "chicken," "eggs," "carrots," and "green peppers," specific recipes such as "oyakodon" (chicken and egg rice bowl) or "stir-fry" are generated. The server also searches an existing recipe database and provides related recipe information. The preferred recipes change depending on the user's emotional state.

[0841] Step 5: Personalize your data

[0842] The server refers to the user's past usage data and personalizes recipes by taking into account the user's preferences, habits, and food expiration dates. The inputs are past usage history, preference information, and food expiration date data. Based on these, the server outputs a recipe optimized for the user. This recipe includes recipes that prioritize the use of ingredients with an approaching expiration date.

[0843] Step 6: Delivery of missing ingredients and equipment

[0844] The server checks whether the user has all the ingredients and equipment required for the generated recipe. If there are any shortages, it automatically generates a delivery request and sends an order to an external supply service. The input is the recipe data and the current list of ingredients and equipment, and the shortages are clearly displayed as output. Based on this shortage information, an order is sent to the delivery service, and the necessary ingredients and equipment are delivered to the user.

[0845] Step 7: Displaying results and cooking assistance

[0846] The server sends the final recipe data, cooking steps, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner, providing text and visual content to guide the user. For example, it displays detailed steps, cooking time, and a list of necessary equipment for making "Oyakodon." It also provides relaxing music and video links depending on the user's emotional state. The input data is the recipe details and guide information, and by displaying and playing this information, the device allows the user to smoothly proceed with cooking.

[0847] (Application example 2)

[0848] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0849] Conventional recipe suggestion systems do not fully consider the use of food in the refrigerator or the user's emotional state, making it difficult to provide a wide variety of recipe suggestions or services that respond to the user's emotions.In addition, in physical stores, there is a lack of services that suggest real-time recipes based on the food items that the user plans to purchase, or that encourage the purchase of necessary food and equipment, making it difficult to improve user satisfaction.

[0850] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating recipes based on the acquired food list and preference information, means for identifying and recognizing foods in the refrigerator through image analysis, means for referencing past usage data and personalizing recipes based on the user's preferences, means for automatically generating and sending a delivery request when necessary foods and utensils are missing, means for recognizing the user's emotional state and adjusting the recipe based on the emotional state, means for photographing foods to be purchased in the store and suggesting recipes, means for suggesting purchases when necessary foods and utensils are missing, and means for providing the user with recipes and cooking instructions. This enables a wide variety of recipes to be suggested based on the user's emotional state and the foods to be purchased, providing a comfortable shopping and cooking experience in a physical store.

[0851] "Acquired food list and preference information" refers to information indicating the food items currently owned by the user and the user's food preferences.

[0852] "Image analysis" is the technique of analyzing digital images to identify specific objects and extract information.

[0853] "Food in refrigerator" means food currently in a user's refrigerated storage unit.

[0854] "Past Usage Data" means historical information and related data generated by a User's use of the System.

[0855] "User Preferences" is information that indicates a user's personal preferences regarding foods and recipes that they like.

[0856] "Personalizing recipes" means adjusting recipes based on the user's individual preferences and past usage data, and suggesting recipes in the most optimal form for each individual user.

[0857] A "delivery request" is order information for delivery of necessary food and equipment to a specified location.

[0858] "User's emotional state" refers to the user's mental and emotional state as determined by analyzing voice and facial expression data.

[0859] "Foods to be purchased" refers to foods that a user intends to purchase in a physical store.

[0860] "Real-time recipe suggestions" means instantly generating and suggesting recipes based on the current situation and user input.

[0861] "Cooking instructions" are the detailed instructions and steps required to prepare a particular food product.

[0862] The system for realizing this invention is mainly composed of a server, a terminal, and a user. Each component and its operation will be explained below.

[0863] 1. Server Functions

[0864] The server has the following functions:

[0865] A means for generating recipes based on the obtained food list and preference information:

[0866] The server generates appropriate recipes based on the food list and cooking preferences entered or photographed by the user, and uses a generative AI model to suggest recipes that best match the ingredients and preferences.

[0867] Image analysis to identify and identify food items in refrigerators:

[0868] The system receives image data of the refrigerator sent by the user and uses an image analysis algorithm (e.g., Google Cloud Vision API) to identify the type and amount of food.

[0869] Ways to personalize recipes based on user preferences using past usage data:

[0870] The server uses past usage history data to suggest recipes tailored to the user's individual preferences, prioritizing recipes that have been popular in the past and ingredients that are frequently used.

[0871] A way to automatically generate and send delivery requests when needed food and equipment are in short supply:

[0872] If the server is running low on ingredients or cooking equipment required for the generated recipe, it will automatically send an order in conjunction with a delivery service and arrange for prompt delivery.

[0873] A way to recognize the user's emotional state and adjust recipes based on that emotional state:

[0874] The server analyzes voice and facial expression data to recognize the user's emotional state. For example, if the user is feeling stressed, it will prioritize recipes that are easy to prepare and have a relaxing effect.

[0875] A way to take a photo of the food you plan to buy in the store and get recipe suggestions:

[0876] The server takes a photo of the food the user plans to buy in a physical store, analyzes the image, and suggests suitable recipes, supporting the user's shopping and increasing their desire to buy.

[0877] Providing suggested purchases of necessary food and equipment if they are in short supply:

[0878] If the user is running low on food or equipment that he or she plans to purchase, the server will indicate this and suggest the purchase.

[0879] 2. Device Features

[0880] The device (e.g., smartphone, tablet) has the following functions:

[0881] Getting user input:

[0882] It offers a variety of input methods, including taking pictures of the refrigerator, entering a food list into a text box, and collecting voice and facial expression data.

[0883] Result display and cooking assistance:

[0884] The system supports cooking by displaying recipe data and cooking instructions sent from the server in an easy-to-understand manner to the user, and also provides relaxing music and video links as needed.

[0885] 3. User Behavior

[0886] Users use the system to perform the following actions:

[0887] Take and upload a picture of the food in your refrigerator.

[0888] Enter your ingredient list and cooking preferences into the text boxes.

[0889] Provides voice and facial expression data.

[0890] Order the necessary ingredients and equipment based on the proposed recipe.

[0891] Examples of specific examples and prompts

[0892] For example, if a user takes a photo of "tomatoes, cheese, pasta" in the refrigerator and the foods they have purchased, and provides the audio data to the app:

[0893] The server will parse the message assuming the following prompt:

[0894] Example prompt sentence:

[0895] "Generate the best recipe from the ingredients you have based on the user's emotional state. The current ingredients are tomatoes, cheese, and pasta."

[0896] This allows for flexible and personalized recipe suggestions tailored to the user's needs, providing a seamless shopping and cooking experience even in physical stores.

[0897] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0898] Step 1:

[0899] The user uses a smartphone or tablet to take pictures of the food in the refrigerator or the food they plan to buy. The user also inputs the food list into a text box and provides voice and facial expression data. The input data consists of image data, text data, and voice and facial expression data.

[0900] Step 2:

[0901] The device sends the collected image data, text data, and voice and facial expression data to the server, where it is entered as user input data.

[0902] Step 3:

[0903] The server uses image analysis algorithms to identify and identify the food in the refrigerator or the food you plan to buy from the image data sent to it. The extracted food data is output. At this stage, specific food names such as "tomato," "cheese," and "pasta" are obtained from the image data.

[0904] Step 4:

[0905] The server analyzes the voice and facial expression data to recognize the user's emotional state. It uses an emotion recognition engine (e.g., Google Cloud Speech-to-Text) to perform the analysis and outputs the user's emotional state. For example, it determines whether the user is relaxed or stressed.

[0906] Step 5:

[0907] The server uses a generative AI model based on the acquired food list and preference information to generate recipes. The input is food data and emotional state data, and the output is a suggested recipe. At this stage, a specific recipe such as "pasta with tomato and cheese" is output.

[0908] Step 6:

[0909] The server references past usage data and personalizes recipes based on the user's preferences. Entering past usage data results in personalized recipes. For example, if a user has previously preferred "tomato pasta," the recipe will be adjusted to take this into account.

[0910] Step 7:

[0911] The server automatically generates and sends a delivery request if the food and equipment required for the generated recipe are missing. The input is the missing food and equipment data, and the output is the generated delivery request. For example, if "cheese" is missing, a request is generated suggesting the purchase of that item.

[0912] Step 8:

[0913] The server sends the final recipe data, cooking instructions, and information on missing foods and equipment to the device. The input is the generated recipe and cooking instructions data, and the output is the information sent to the device. Specifically, the step-by-step cooking instructions and instructions for purchasing "cheese" are displayed.

[0914] Step 9:

[0915] The device finally displays the recipe and cooking instructions to the user, and provides relaxing music and video links as needed.The user can then cook based on the displayed information and enjoy the suggested recipe.

[0916] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0917] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0918] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0919] [Third embodiment]

[0920] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0921] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0922] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0923] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0924] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0925] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0926] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0927] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0928] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0929] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0930] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0931] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0932] The present invention provides a system for solving the problems of cooking in a rut and food waste in busy daily lives. Below, an embodiment of this system is described.

[0933] 1. Getting User Input

[0934] To help users efficiently utilize the ingredients in their refrigerator and enjoy a wide variety of dishes, the device first acquires input data. The device provides users with the options to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their favorite dishes. This allows the device to collect information about the user.

[0935] 2. Image Analysis and Data Conversion

[0936] The image data of the refrigerator sent by the device is received by the server. The server uses an image analysis algorithm to identify and identify the ingredients present in the refrigerator. At the same time, the list of ingredients and cooking preferences are also analyzed and converted into text data. For example, ingredients such as "carrots," "bell peppers," "chicken," and "eggs" can be recognized from the image and recorded as text data.

[0937] 3. Recipe Creation and Search

[0938] The server generates recipes based on the acquired ingredient list and preference information. Using a generative AI model, it suggests recipes that make the most of the ingredients the user has. For example, if a user has chicken, eggs, carrots, and green peppers, it will suggest recipes such as oyakodon (chicken and egg rice bowl) or stir-fry. It also references an existing recipe database to search for and provide related recipe information.

[0939] 4. Personalizing your data

[0940] The server refers to the user's past usage data and personalizes recipes based on the user's preferences and habits. It also takes into account the expiration dates of ingredients and makes suggestions to minimize food waste. For example, it can prevent food waste by suggesting recipes that prioritize ingredients with a short expiration date.

[0941] 5. Delivery of missing ingredients and equipment

[0942] If the server does not have the ingredients or equipment required for the generated recipe on hand, it automatically generates a delivery request and sends an order to an external supply service. The delivery service quickly arranges for the missing ingredients and equipment and delivers them to the user. For example, if there is a shortage of "chicken," that information is sent to the delivery service and arrangements are made to procure it.

[0943] 6. Results display and cooking assistance

[0944] The server sends the final recipe data, cooking instructions, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner to the user, providing text and visual content to guide the user through the cooking methods and procedures. For example, the device displays detailed instructions, cooking time, and a list of necessary equipment for making "Oyakodon."

[0945] Specific examples

[0946] Example 1: Upload an image of a refrigerator

[0947] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app.

[0948] 2. The server analyzes the image and recognizes "carrots," "bell peppers," "chicken," and "eggs."

[0949] 3. The server generates a recipe for "Oyakodon" based on these ingredients and sends the cooking instructions to the terminal.

[0950] 4. The device displays cooking instructions to the user.

[0951] Example 2: Enter a list of ingredients

[0952] 1. A user types "tomato, cheese, pasta" into a text box.

[0953] 2. The server analyzes this, searches for related recipes, and suggests "pasta with tomatoes and cheese."

[0954] 3. The server generates a recipe that takes expiration dates into account and sends the details to the device.

[0955] 4. The device displays the recipe and cooking instructions to the user.

[0956] In this way, the present invention can effectively utilize leftover ingredients in the refrigerator, preventing daily cooking from becoming monotonous and providing users with a fresh and healthy diet.

[0957] The processing flow will be explained below.

[0958] Step 1:

[0959] A user launches the application using a device such as a smartphone or tablet, which then provides the user with the option to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their preferred dish.

[0960] Step 2:

[0961] Users can take a picture of their refrigerator and upload it to the application, or enter their ingredient list and cooking preferences into a text box.

[0962] Step 3:

[0963] The terminal sends the user's input data (image data or text data) to the server.

[0964] Step 4:

[0965] The server uses an image analysis algorithm to analyze the image data received from the refrigerator. Specifically, it applies a food recognition model to identify ingredients from the image. This allows ingredients such as "carrots," "bell peppers," "chicken," and "eggs" to be identified and classified.

[0966] Step 5:

[0967] The server performs text analysis on the user's input of ingredients and cooking preferences to extract ingredients and cooking genres, which are also used to generate recipes.

[0968] Step 6:

[0969] The server uses the generative AI model to generate customized recipes based on the user's food list and preferences, and also references existing recipe databases such as Classy to search and retrieve related recipe information.

[0970] Step 7:

[0971] The server references the user's past usage data, taking into account information such as past favorite recipes and commonly used ingredients, and personalizes recipes based on the user's preferences.

[0972] Step 8:

[0973] The server creates a list of ingredients and equipment needed for the generated recipe, checking the user's ingredient list to ensure all ingredients are available.

[0974] Step 9:

[0975] The server adjusts recipes taking into account the expiration dates of ingredients, suggesting recipes that prioritize ingredients with close expiration dates and preventing food waste.

[0976] Step 10:

[0977] If the server is running low on ingredients or equipment, it automatically generates a delivery request, which is then sent to an external supply service (e.g., an online shopping site).

[0978] Step 11:

[0979] The server receives confirmation from the delivery service and sends a notification to the user, for example, "Your chicken will arrive in 30 minutes."

[0980] Step 12:

[0981] The server sends the final recipe data, cooking instructions, and information on missing ingredients and utensils to the terminal.

[0982] Step 13:

[0983] The device receives information and displays it to the user in an easy-to-understand manner, such as providing recipe steps, necessary equipment, and links to cooking videos.

[0984] Step 14:

[0985] The user begins cooking according to the recipe displayed on the device, with the help of a timer and additional tips provided by the app.

[0986] Step 15:

[0987] The device records the user's cooking progress, sends the data back to the server, and stores it in a database for future personalization.

[0988] Example 1

[0989] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0990] In today's busy lifestyles, cooking at home can easily become monotonous, and food in the refrigerator is often underutilized and thrown away. While this problem differs from household to household, the following issues need to be resolved: First, to efficiently utilize the food in the refrigerator and provide nutritionally balanced meals. Second, to suggest personalized recipes that take into account the user's preferences and past cooking history. Third, to assist the user in cooking by quickly arranging for ingredients and utensils that are in short supply.

[0991] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0992] In this invention, the server includes means for generating recipes based on the acquired ingredient list and preference information, means for identifying and identifying ingredients in the refrigerator through image analysis, means for referencing past usage data and personalizing recipes based on the user's preferences, means for automatically generating and sending a delivery request when necessary ingredients and cooking utensils are in short supply, means for providing the user with recipes and cooking instructions, means for generating recipes based on input data using a generative AI model, and means for automatically generating prompt statements, inputting them into the AI ​​model, and generating recipes based on the results. This makes it possible to efficiently use ingredients in the refrigerator, provide personalized recipes that suit the user's preferences, and minimize food waste.

[0993] The "obtained ingredient list" refers to the information about ingredients in the refrigerator entered by the user, including image analysis results and text input data.

[0994] "Preference information" is data about a user's preferences based on their cooking preferences, past usage history, etc.

[0995] The "means for generating recipes" is a function that creates new recipes using a generative AI model and recipe database based on the acquired ingredient list and preference information.

[0996] "Image analysis" is a technique that processes image data to identify ingredients in the refrigerator. It includes image recognition algorithms and machine learning models.

[0997] "Means for identification" refers to the function of identifying and specifying ingredients based on certain criteria from the information obtained by image analysis.

[0998] "Past Usage Data" refers to all data from a user's previous use of the system, including information about recipes selected and ingredients used.

[0999] "Personalization" refers to a function that suggests individually appropriate recipes based on the user's past usage data and preferences.

[1000] "Delivery Request" refers to a request to send an order to an external supply service when necessary ingredients or cooking equipment are in short supply.

[1001] "Means for providing cooking instructions" is a function that clearly provides specific steps and information on necessary equipment when a user cooks a dish.

[1002] A "generative AI model" is an artificial intelligence model used for complex data processing such as image analysis and recipe generation, as well as natural language processing.

[1003] A "prompt sentence" is a text sentence that expresses a question or request that is input to a generative AI model, and the AI ​​generates a response based on this.

[1004] The present invention is a system that solves the problems of cooking in a rut and food waste in busy daily lives. This system efficiently utilizes ingredients in the refrigerator and suggests a wide variety of dishes, providing users with a fresh and healthy diet. Detailed embodiments for implementing this system are described below.

[1005] Getting User Input

[1006] Users use a dedicated application on their smartphone or PC to input information about the ingredients in their refrigerator. The input method is as follows:

[1007] 1. Take a picture of the ingredients in your refrigerator and upload it.

[1008] 2. Enter the ingredients list in the text box.

[1009] 3. Use the drop-down menu to select your preferred cuisine.

[1010] Image analysis and data conversion

[1011] The device sends image data, text data, or data on the user's favorite dishes to the server. The server then uses an image analysis algorithm (using, for example, OpenCV or TensorFlow) to identify the ingredients in the refrigerator. For example, it identifies "carrots," "bell peppers," "chicken," and "eggs" from the image and saves them as text data. This allows the server to obtain a list of ingredients and the user's favorite dishes.

[1012] Recipe Generation

[1013] The server generates a recipe using a generative AI model (for example, GPT-3 or a similar natural language processing model) based on the acquired ingredient list and preference information. Specifically, the server automatically generates a prompt sentence and inputs it into the generative AI model. For example, a prompt sentence such as "What recipes use chicken, eggs, carrots, and green peppers?" is generated and input into the AI ​​model. The recipe is generated based on the results obtained from the AI ​​model.

[1014] Data Personalization

[1015] The server refers to the user's past usage data and proposes personalized recipes that take into account the user's preferences and the expiration dates of ingredients, thereby prioritizing the use of ingredients with an approaching expiration date, thereby minimizing food waste.

[1016] Delivery of missing ingredients and cooking equipment

[1017] If the server is running low on ingredients or cooking utensils required for the generated recipe, it automatically generates a delivery request and sends it to an external supply service. For example, if there is a shortage of chicken, the server sends that information to the delivery service and quickly arranges for the missing ingredients or cooking utensils.

[1018] Result display and cooking assistance

[1019] The server sends the final recipe data, cooking instructions, and information on missing ingredients and cooking utensils to the device. The device then displays this information in an easy-to-understand manner to the user, guiding them through the cooking methods and procedures. For example, the device displays detailed instructions for making an "oyakodon," cooking time, and a list of necessary utensils. It also displays text and visual content (images and videos) to guide the user through the cooking process.

[1020] Specific examples

[1021] Example 1: Upload an image of a refrigerator

[1022] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app.

[1023] 2. The server analyzes the image and identifies "carrots," "bell peppers," "chicken," and "eggs."

[1024] 3. The server generates a recipe for "Oyakodon" based on these ingredients and sends the cooking instructions to the terminal.

[1025] 4. The device displays cooking instructions to the user.

[1026] Example 2: Enter a list of ingredients

[1027] 1. A user types "tomato, cheese, pasta" into a text box.

[1028] 2. The server analyzes this, searches for related recipes, and suggests "pasta with tomatoes and cheese."

[1029] 3. The server generates a recipe that takes expiration dates into account and sends the details to the device.

[1030] 4. The device displays the recipe and cooking instructions to the user.

[1031] In this way, users can efficiently utilize the ingredients in their refrigerator and enjoy fresh and healthy meals without getting bored with their daily cooking routine.

[1032] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1033] Step 1: Getting User Input

[1034] Users launch the application on their smartphone or PC and enter information about the ingredients in their refrigerator by taking a picture of the refrigerator and uploading it, entering the list of ingredients in a text box, or selecting the option to choose their favorite dish.

[1035] Input: Image data of the contents of the refrigerator, text data, or favorite food information

[1036] Output: User input data (image data, text data, favorite food information)

[1037] Step 2: Image analysis and data conversion

[1038] The terminal transmits the acquired user input data to the server.

[1039] The server processes the received image data using an image analysis algorithm (e.g., OpenCV or TensorFlow) to identify the ingredients in the refrigerator. For example, it identifies "carrots," "bell peppers," "chicken," and "eggs" from the image.

[1040] Input: User-entered data (image data, text data, favorite food information)

[1041] Data processing: Identification and classification of ingredients using image analysis

[1042] Output: Identified ingredients list (text data)

[1043] Step 3: Generate the recipe

[1044] Based on the ingredient list and preference information acquired by the server, a recipe is generated using a generative AI model (e.g., GPT-3). The server generates a prompt sentence and inputs it into the generative AI model. For example, a prompt sentence such as "What is a recipe that uses chicken, eggs, carrots, and green peppers?" is generated and input into the AI ​​model.

[1045] Input: Ingredient list, preference information

[1046] Data processing: Recipe generation using generative AI models

[1047] Output: Suggested recipe (text data)

[1048] Step 4: Personalize your data

[1049] The server references the user's past usage data and generates personalized recipes that take into account the user's preferences and the expiration dates of ingredients. Specifically, it suggests recipes that prioritize ingredients with an approaching expiration date.

[1050] Input: Ingredient list, preference information, past usage data

[1051] Data processing: Consideration of preference analysis and expiration date data

[1052] Output: Personalized recipe (text data)

[1053] Step 5: Delivery of missing ingredients and cooking equipment

[1054] The server creates a list of ingredients and cooking tools needed based on the recipe. If any of the ingredients are missing, it automatically generates a delivery request and sends it to an external supply service. For example, if chicken is in short supply, it requests a delivery service to procure it.

[1055] Input: Personalized recipe, information about ingredients and cooking equipment the user has on hand

[1056] Data processing: Identifying missing ingredients and cooking utensils, generating automatic delivery requests

[1057] Output: Delivery request

[1058] Step 6: Displaying results and cooking assistance

[1059] The server sends the final recipe data, along with information on missing ingredients and cooking utensils, to the device. The device then displays this information in an easy-to-understand manner to the user, guiding them through the cooking method and steps. For example, it displays detailed instructions, cooking time, and a list of necessary utensils for making "Oyakodon."

[1060] Input: Final recipe data, cooking instructions, missing ingredients and utensils information

[1061] Data processing: Format conversion of cooking instructions and necessary information

[1062] Output: Cooking guide display for the user (text and visual content)

[1063] Through these steps, the system can efficiently utilize the ingredients in the refrigerator and provide users with a fresh and healthy diet.

[1064] (Application example 1)

[1065] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1066] Conventional food management systems do not adequately suggest recipes that efficiently utilize ingredients in the refrigerator, resulting in food waste. Furthermore, recipe suggestions that take into account the user's preferences and past cooking history are limited. In particular, the lack of a convenient way to provide cooking instructions or delivery of missing ingredients in busy daily lives has been a major problem for users.

[1067] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1068] In this invention, the server includes: means for generating recipes based on the acquired food list and preference information; means for identifying and recognizing foods in the refrigerator through image analysis; means for referencing past usage data and personalizing recipes based on the user's preferences; means for automatically generating and sending delivery requests when necessary foods and utensils are missing; means for providing recipes and cooking instructions to the user; means for processing food images taken with a smartphone; means for automatically recognizing the acquired food list using an image analysis algorithm; means for using a generative AI model to generate and suggest optimal recipes based on the acquired food list; and means for transmitting the missing list to a delivery service. This enables effective use of ingredients in the refrigerator, reduction of food waste, personalized recipe suggestions for users, and easy delivery ordering.

[1069] - "Food List" refers to a list of ingredients owned by the user.

[1070] "Preference information" refers to information about the cuisine and taste preferences of a user that has been selected in the past.

[1071] The "means for generating recipes" is a function that automatically creates recipes showing cooking methods and ingredients based on the acquired food list and preference information.

[1072] "Image analysis" refers to techniques and processes that process digital image data to extract specific information.

[1073] "Means for identifying and discerning food in a refrigerator" refers to a technology that uses image analysis to recognize and identify food items present in a refrigerator.

[1074] "Past usage data" refers to historical information such as ingredients used by the user, recipes selected, and ingredients ordered.

[1075] "Means for personalizing recipes based on user preferences" refers to technology that references past usage data and suggests recipes tailored to the user's specific preferences.

[1076] "Means for automatically generating and sending delivery requests" refers to a function that automatically detects when necessary food or equipment is missing and sends an order to an external delivery service.

[1077] "Means for providing recipes and cooking instructions" refers to a function that displays or communicates suggested recipes and their cooking instructions to the user.

[1078] The "means for processing food images taken by a smartphone" refers to the process of receiving image data taken by a user, analyzing it, and extracting a list of foods to be used.

[1079] An "image analysis algorithm" is a set of mathematical models or programs that process image data to recognize specific objects or text information.

[1080] A "generative AI model" is an artificial intelligence model that uses machine learning to generate new recipes and information from data.

[1081] A "delivery service" is a service organization that delivers ingredients and equipment to users.

[1082] The present invention is a system that efficiently utilizes ingredients in a refrigerator, provides a wide variety of dishes, and prevents food waste. This system starts when a user takes a photo of the ingredients in the refrigerator with their smartphone and uploads it to an application.

[1083] Hardware and software used

[1084] Hardware:

[1085] Smartphone: Taking pictures of ingredients and using applications.

[1086] Server: Image analysis, recipe generation, and delivery request processing.

[1087] Delivery service computer system: Arranges delivery of ingredients and cooking equipment.

[1088] software:

[1089] Flask: A web application framework in Python.

[1090] PIL (Pillow): Image processing library.

[1091] food_recognition: Food recognition algorithm.

[1092] recipe_generation: Recipe generation algorithm.

[1093] delivery_service: Third-party delivery service API.

[1094] Processing flow

[1095] 1. Getting User Input

[1096] Users take a photo of the ingredients in their refrigerator with their smartphone and upload the image to the application.

[1097] Alternatively, you're given the option to manually enter the ingredient list into a text box.

[1098] Information about the user's preferences and past cooking tastes can also be entered.

[1099] 2. Image Analysis and Data Conversion

[1100] The server receives the image data sent from the smartphone and reads the image data using Pillow.

[1101] Ingredients are identified using an image analysis algorithm (food_recognition), and the information is converted into text data.

[1102] For example, "tomato," "cheese," and "beef" can be identified from an image of a refrigerator and recorded as text data.

[1103] 3. Recipe Generation

[1104] A generative AI model (recipe_generation) is used to generate recipes based on the acquired ingredient list and user preference information.

[1105] For example, we suggest a recipe for Bolognese pasta using tomatoes, cheese, and beef.

[1106] 4. Creating a delivery request

[1107] The server checks the ingredients and equipment required for the generated recipe, and if any are missing, it automatically generates a delivery request for them.

[1108] It calls the delivery service's API and arranges for the necessary ingredients and equipment to be delivered to the user.

[1109] 5. Results display and cooking assistance

[1110] The server sends the final recipe data, cooking instructions, and information on missing ingredients and utensils to the smartphone.

[1111] The smartphone displays this information in an easy-to-understand way for the user, guiding them through the cooking process with visual and textual content.

[1112] Specific examples

[1113] Example 1: Uploading food images

[1114] Users take a picture of the inside of their refrigerator with their smartphone and upload it to the app.

[1115] The server analyzes the image and recognizes "carrots," "bell peppers," "chicken," and "eggs."

[1116] The server uses a generative AI model to generate a recipe for "Oyakodon" based on these ingredients and sends the cooking instructions to a smartphone.

[1117] The smartphone displays cooking instructions to the user.

[1118] Example 2: Ordering delivery of ingredients that are in short supply

[1119] A user uploads a picture of their refrigerator and discovers that they are out of "chicken."

[1120] The server generates a delivery request and places an order for "chicken" through the delivery service API.

[1121] The delivery service arranges for the chicken to be delivered to the user.

[1122] Example 3: Example of a prompt statement

[1123] Prompt: "How can I build a system that allows you to upload a picture of the ingredients in your refrigerator, suggest recipes based on the images, and order delivery for any missing ingredients? What are the basic features this system should have?"

[1124] As a result, this system can make effective use of ingredients in the refrigerator, reduce food waste, and provide users with personalized recipe suggestions and delivery services.

[1125] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1126] Step 1:

[1127] The user takes a photo of the food in the refrigerator with their smartphone and uploads the image to the application.

[1128] Specific actions

[1129] Users use their smartphone camera to take a picture of the ingredients in their refrigerator and upload the image to a dedicated app, which also offers the option to input user preference information and allows users to provide past usage data.

[1130] Input and Output

[1131] Input: Image data of ingredients, preference information

[1132] Output: Image data and preference information sent to the server

[1133] Step 2:

[1134] The server receives the image data and loads the image using Pillow.

[1135] Specific actions

[1136] The server receives the uploaded image data and reads the image using the Pillow image processing library, thereby obtaining the digital image data.

[1137] Input and Output

[1138] Input: Image data sent from a smartphone

[1139] Output: Loaded image data

[1140] Step 3:

[1141] The server uses an image analysis algorithm (food_recognition) to identify ingredients and convert them into text data.

[1142] Specific actions

[1143] The server analyzes the received image data using the food_recognition algorithm, identifies the ingredients in the image, converts the identified ingredients into text data, and creates a list.

[1144] Input and Output

[1145] Input: Image data

[1146] Output: Ingredient list (text data)

[1147] Step 4:

[1148] The server uses a generative AI model (recipe_generation) to generate a recipe based on the ingredient list and preference information.

[1149] Specific actions

[1150] The server uses a generative AI model to create an appropriate recipe based on the acquired ingredient list and the user's preferences. The generated recipe also includes specific cooking steps.

[1151] Input and Output

[1152] Input: Ingredient list, preference information

[1153] Output: Generated recipe and cooking instructions

[1154] Step 5:

[1155] The server checks for missing ingredients and equipment based on the generated recipe and generates a delivery request.

[1156] Specific actions

[1157] The server compares all ingredients included in the generated recipe with the user's current inventory, identifies any missing ingredients or utensils, and generates and sends a request to an external delivery service API.

[1158] Input and Output

[1159] Input: Generated recipe, ingredient list

[1160] Output: Delivery request

[1161] Step 6:

[1162] The server sends the final recipe data, cooking instructions, and delivery request results to the smartphone.

[1163] Specific actions

[1164] The server then sends the generated recipe data, cooking instructions, and delivery results for missing ingredients and utensils to a smartphone, where users can check the information using the smartphone app.

[1165] Input and Output

[1166] Input: Final recipe data, cooking instructions, delivery request results

[1167] Output: Recipe data and cooking instructions displayed on a smartphone

[1168] Through these steps, the system can effectively utilize the ingredients in the refrigerator and provide personalized recipe suggestions and delivery services.

[1169] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1170] The present invention is a system that efficiently utilizes ingredients and utensils that a user has, and also suggests recipes and cooking methods according to the user's emotional state. The following describes an embodiment of this system.

[1171] 1. Getting User Input

[1172] To help users efficiently utilize the ingredients in their refrigerator and enjoy a wide variety of dishes, the device first acquires input data. The device provides users with the option to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their preferred dishes. Additionally, the device also has the ability to collect voice and facial expression data to recognize the user's emotions.

[1173] 2. Image Analysis and Data Conversion

[1174] The image data of the refrigerator sent by the device is received by the server. The server uses an image analysis algorithm to identify and identify the ingredients present in the refrigerator. At the same time, the list of ingredients and cooking preferences are also analyzed and converted into text data. For example, ingredients such as "carrots," "bell peppers," "chicken," and "eggs" can be recognized from the image and recorded as text data.

[1175] 3. Emotional Recognition

[1176] The device collects the user's voice and facial expression data and sends it to the server. The server then uses an emotion engine to analyze and recognize the user's emotions from this data. For example, it analyzes changes in voice tone and facial expressions to determine whether the user is relaxed or stressed.

[1177] 4. Recipe Creation and Search

[1178] The server generates recipes based on the acquired ingredient list and preference information. Using a generative AI model, it suggests recipes that make the most of the ingredients the user has. It also references an existing recipe database to search for and provide related recipe information. For example, if a user has chicken, eggs, carrots, and green peppers, it will suggest recipes such as oyakodon (chicken and egg rice bowl) or stir-fry. Depending on the user's emotional state, it also prioritizes suggestions of very simple recipes and dishes with a relaxing effect.

[1179] 5. Personalization of Data

[1180] The server refers to the user's past usage data and personalizes recipes based on the user's preferences and habits. It also takes into account the expiration dates of ingredients and makes suggestions to minimize food waste. For example, it can prevent food waste by suggesting recipes that prioritize ingredients with a short expiration date.

[1181] 6. Delivery of missing ingredients and equipment

[1182] If the server does not have the ingredients or equipment required for the generated recipe on hand, it automatically generates a delivery request and sends an order to an external supply service. The delivery service quickly arranges for the missing ingredients and equipment and delivers them to the user. For example, if there is a shortage of "chicken," that information is sent to the delivery service and arrangements are made to procure it.

[1183] 7. Results display and cooking assistance

[1184] The server sends the final recipe data, cooking instructions, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner to the user, providing text and visual content to guide the user through the cooking methods and procedures. For example, the device may display detailed instructions, cooking time, and a list of necessary equipment for making "Oyakodon." Furthermore, the device may also provide relaxing music and video links depending on the user's emotional state.

[1185] Specific examples

[1186] Example 1: Uploading images and audio data of a refrigerator

[1187] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app, which also collects audio data.

[1188] 2. The server analyzes the image and recognizes "carrot," "bell pepper," "chicken," and "egg." At the same time, it analyzes the voice data and recognizes the user's emotional state.

[1189] 3. The server generates a recipe for "Oyakodon" based on this information and sends the cooking instructions to the device. If the user is feeling stressed, it also suggests relaxing music.

[1190] 4. The device displays and plays cooking instructions and relaxing music to the user.

[1191] Example 2: Input ingredients list and facial expression data

[1192] 1. The user enters "tomato, cheese, pasta" into the text box, and facial expression data is collected using the camera.

[1193] 2. The server analyzes this, searches for related recipes, and suggests "Tomato and Cheese Pasta." It also determines from facial expression data that the user is relaxed.

[1194] 3. The server generates a recipe that takes expiration dates into account and sends the details to the device.

[1195] 4. The device displays the recipe and cooking instructions to the user.

[1196] In this way, the present invention effectively utilizes leftover ingredients in the refrigerator and suggests recipes that take into consideration the user's emotional state, preventing cooking from becoming monotonous, reducing food waste, and providing a richer diet.

[1197] The processing flow will be explained below.

[1198] Step 1:

[1199] A user launches the application using a device such as a smartphone or tablet. The device provides the user with the options to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their preferred dish. It also prompts them to collect voice data and capture facial expression data.

[1200] Step 2:

[1201] Users take a picture of their refrigerator and upload it to the application, and are also asked to record a voice message or enter facial expression data via the camera to ascertain its emotional state.

[1202] Step 3:

[1203] The terminal sends the user's input data (image data, voice data, facial expression data, or text data) to the server.

[1204] Step 4:

[1205] The server analyzes the image data of the refrigerator it receives using an image analysis algorithm. Specifically, it uses a food recognition model to identify ingredients from the image. As a result, ingredients such as "carrots," "bell peppers," "chicken," and "eggs" in the refrigerator are identified.

[1206] Step 5:

[1207] The server analyzes the user's ingredient list and cooking preferences received as text data and extracts the ingredient list and cooking categories.

[1208] Step 6:

[1209] The server uses an emotion engine to analyze the transmitted voice and facial expression data and recognize the user's emotional state. For example, it can determine whether the user is relaxed or stressed based on changes in voice tone and facial expression.

[1210] Step 7:

[1211] The server generates recipes based on the acquired ingredient list and the user's recognized emotional state. Using a generative AI model, the server suggests recipes that make the most of the ingredients the user has. The server then adjusts the recipes according to the user's emotional state. For example, if the user is tired, the server suggests simple recipes that have a relaxing effect.

[1212] Step 8:

[1213] The server uses past usage data to personalize recipes based on the user's preferences and habits, and also takes into account expiration dates of ingredients, prioritizing recipes that minimize food waste.

[1214] Step 9:

[1215] The server creates a list of ingredients and tools required for the generated recipe, referencing the user's ingredient list to see what is available and what is missing.

[1216] Step 10:

[1217] If the server is missing ingredients or equipment, it automatically generates a delivery request, which is then sent to an external supply service.

[1218] Step 11:

[1219] The server receives confirmation from the delivery service and notifies the user when the missing ingredients or equipment will arrive, for example, "The chicken will arrive in 30 minutes."

[1220] Step 12:

[1221] The server sends the final recipe data, cooking instructions, and information on missing ingredients and utensils to the terminal.

[1222] Step 13:

[1223] The device displays the information received in an easy-to-understand manner to the user, providing recipe steps, necessary utensils, and links to cooking videos through text and visual content. It also displays relaxing music and videos based on the user's emotional state.

[1224] Step 14:

[1225] The user begins cooking by following the recipe displayed on the device, with the help of a timer and additional tips provided by the app.

[1226] Step 15:

[1227] The device records the user's cooking progress, sends the data back to the server, and stores it in a database for future personalization.

[1228] Example 2

[1229] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1230] Currently, many households are unable to efficiently utilize the ingredients in their refrigerators, resulting in food waste. It is also difficult for systems to suggest dishes that reflect the user's emotional state, which can reduce meal satisfaction. Furthermore, if the necessary ingredients or cooking utensils are in short supply, it takes time to procure them, making it difficult to start cooking quickly. There is a need to solve these issues and provide users with an efficient and satisfying cooking experience.

[1231] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1232] In this invention, the server includes means for receiving and analyzing user input as text data and image data, means for identifying and identifying foods in the refrigerator through image analysis, means for generating recipes based on the acquired food list and preference information, means for analyzing the user's emotional state using voice and facial expression data, means for suggesting recipes according to the user's emotional state, means for personalizing recipes based on the user's preferences by referencing past usage data, means for automatically generating and sending a delivery request when necessary foods and utensils are in short supply, means for providing the user with recipes and cooking instructions, and means for generating recipes using a generative AI model. This makes it possible to efficiently use ingredients in the refrigerator, suggest recipes according to the user's emotional state, and quickly procure the necessary ingredients and utensils.

[1233] "User input" refers to text data, image data, voice data, and facial expression data that a user provides to the system.

[1234] "Text data" refers to data entered by the user in the form of text, such as a list of ingredients or preference information.

[1235] "Image data" refers to images of food and other visual information in the refrigerator taken with a camera and uploaded to the system.

[1236] "Audio Data" refers to audio information collected for the purpose of analyzing a user's emotional state.

[1237] "Facial expression data" refers to emotional information extracted from images of a user's facial expressions captured by a camera.

[1238] "Image analysis" refers to the process of analyzing transmitted image data to identify and identify the food items in the refrigerator.

[1239] A "food list" refers to data that records in text format all the food in your refrigerator.

[1240] "Preference information" refers to information that represents a user's preferences for food and cooking.

[1241] "Emotional state" refers to the user's current psychological and emotional state, analyzed from their voice and facial expression data.

[1242] A "recipe" is a document or piece of information that lists the ingredients and steps needed to prepare a particular dish.

[1243] "Generative AI model" refers to an algorithmic model that uses artificial intelligence to generate new recipes.

[1244] "Past usage data" refers to data that records a user's system usage history.

[1245] "Food waste" refers to food that is discarded without being consumed.

[1246] "Delivery Request" means an order request sent to an external supply service to procure needed food and / or equipment.

[1247] "Personalization" refers to the process of providing users with optimized suggestions and services based on their past usage history and preferences.

[1248] The present invention is a system that efficiently utilizes the ingredients and cooking utensils that a user has, and also suggests recipes and cooking methods that correspond to the user's emotional state. Specific implementation methods of this system are described in detail below.

[1249] Getting User Input

[1250] Users input ingredient information and their favorite dishes using a smartphone or PC. The device provides users with the ability to take and upload images of their refrigerator, a text box for entering a list of ingredients, and check boxes and drop-down menus for selecting their favorite dishes. The device also collects user emotional data through voice input and facial recognition using a camera.

[1251] Image analysis and data conversion

[1252] The server receives the image data of the refrigerator sent by the device and uses image analysis algorithms (e.g., OpenCV or TensorFlow) to identify and distinguish the ingredients in the refrigerator. For example, it recognizes "carrots," "bell peppers," "chicken," and "eggs" from the image of the refrigerator and records them as text data. In addition, it analyzes the text list of ingredients and cooking preferences entered by the user and stores them in a database.

[1253] Emotion recognition

[1254] The device sends the collected voice and facial expression data to a server, which then analyzes it using an emotion analysis engine (e.g., IBM Watson or Microsoft Azure's emotion analysis API). This determines whether the user is relaxed or stressed. The analysis results are then recorded in a database.

[1255] Recipe creation and search

[1256] The server uses a generative AI model (e.g., GPT-4) to generate new recipes based on the acquired ingredient list and emotional data. For example, if a user has chicken, eggs, carrots, and green peppers, it will suggest recipes such as oyakodon (chicken and egg rice bowl) or stir-fry. It will also search an existing recipe database to provide related recipe information. If the user is feeling stressed, it will prioritize suggestions of easy recipes and dishes that have a relaxing effect.

[1257] Data Personalization

[1258] The server refers to the user's past usage data and personalizes recipes based on the user's preferences. For example, if there is an ingredient that is close to its expiration date, it can suggest a recipe that prioritizes using that ingredient, thereby reducing food waste.

[1259] Delivery of missing ingredients and equipment

[1260] The server checks whether the user has the ingredients and equipment required for the generated recipe. If they are in short supply, it automatically generates a delivery request and sends an order to an external supply service. For example, if there is a shortage of "chicken," that information is sent to the delivery service and arrangements are made for timely procurement.

[1261] Result display and cooking assistance

[1262] The server sends the final recipe data, cooking instructions, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner to the user, providing text and visual content to guide the user through the cooking method and steps. For example, it may display detailed instructions, cooking time, and a list of necessary equipment for making "Oyakodon." It may also provide relaxing music or video links depending on the user's emotional state.

[1263] Specific examples

[1264] Example 1: Uploading images and audio data of a refrigerator

[1265] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app, which also collects audio data.

[1266] 2. The device sends image and audio data to the server.

[1267] 3. The server analyzes the image and recognizes "carrot," "bell pepper," "chicken," and "egg." It also analyzes the voice data to understand the user's emotional state.

[1268] 4. The server generates a recipe for "Oyakodon" based on the recognized ingredients and emotional data. If it determines that the user is feeling stressed, it selects music with a relaxing effect.

[1269] 5. The server sends the generated recipe and music information to the device.

[1270] 6. The device displays and plays cooking instructions and relaxing music to the user.

[1271] Example 2: Input ingredients list and facial expression data

[1272] 1. The user enters "tomato, cheese, pasta" into the text box, and facial expression data is collected using the camera.

[1273] 2. The device sends the text and facial expression data to the server.

[1274] 3. The server analyzes the text data to search for relevant recipes and determines whether the user is relaxed based on facial expression data.

[1275] 4. The server generates a recipe for "Tomato and Cheese Pasta" and creates a recipe that takes into account the expiration date.

[1276] 5. The server sends the generated recipe to the device.

[1277] 6. The device displays the recipe and cooking instructions to the user.

[1278] In this way, the present invention effectively utilizes leftover ingredients in the refrigerator and suggests recipes that take into consideration the user's emotional state, preventing cooking from becoming monotonous, reducing food waste, and providing a richer diet.

[1279] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1280] Step 1: Getting User Input

[1281] Users open the app on their smartphone or PC and enter information about ingredients in their refrigerator and their favorite dishes, or take and upload images. Voice input and facial expression data from the camera are also collected. Image data, text data, voice data, and facial expression data are obtained as input, and this input data is sent to the system. The device collects this data, converts it into an appropriate format, and sends it to the server.

[1282] Step 2: Image analysis and data conversion

[1283] The server receives the image data of the refrigerator sent from the device and uses image analysis algorithms (e.g., OpenCV or TensorFlow) to identify the ingredients in the refrigerator. In this process, image data is input and converted into text data. For example, it identifies "carrots," "bell peppers," "chicken," and "eggs" from the image of the refrigerator and outputs them as text data. The server also analyzes the text list of ingredients and cooking preferences entered by the user and stores them in a database.

[1284] Step 3: Recognize emotions

[1285] The server receives the voice and facial expression data sent from the device and analyzes the user's emotional state using an emotion analysis engine (e.g., IBM Watson or Microsoft Azure's emotion analysis API). Voice and facial expression data are received as input, and they are analyzed to output information about the user's emotional state (relaxed, stressed, etc.). The analysis results are stored in a database.

[1286] Step 4: Creating and searching recipes

[1287] The server generates new recipes using a generative AI model (e.g., GPT-4) based on the acquired ingredient list, emotional data, and preference information. In this process, the ingredient list and emotional state are input as text data, and recipe suggestions based on them are output. For example, if a user has "chicken," "eggs," "carrots," and "green peppers," specific recipes such as "oyakodon" (chicken and egg rice bowl) or "stir-fry" are generated. The server also searches an existing recipe database and provides related recipe information. The preferred recipes change depending on the user's emotional state.

[1288] Step 5: Personalize your data

[1289] The server refers to the user's past usage data and personalizes recipes by taking into account the user's preferences, habits, and food expiration dates. The inputs are past usage history, preference information, and food expiration date data. Based on these, the server outputs a recipe optimized for the user. This recipe includes recipes that prioritize the use of ingredients with an approaching expiration date.

[1290] Step 6: Delivery of missing ingredients and equipment

[1291] The server checks whether the user has all the ingredients and equipment required for the generated recipe. If there are any shortages, it automatically generates a delivery request and sends an order to an external supply service. The input is the recipe data and the current list of ingredients and equipment, and the shortages are clearly displayed as output. Based on this shortage information, an order is sent to the delivery service, and the necessary ingredients and equipment are delivered to the user.

[1292] Step 7: Displaying results and cooking assistance

[1293] The server sends the final recipe data, cooking steps, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner, providing text and visual content to guide the user. For example, it displays detailed steps, cooking time, and a list of necessary equipment for making "Oyakodon." It also provides relaxing music and video links depending on the user's emotional state. The input data is the recipe details and guide information, and by displaying and playing this information, the device allows the user to smoothly proceed with cooking.

[1294] (Application example 2)

[1295] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1296] Conventional recipe suggestion systems do not fully consider the use of food in the refrigerator or the user's emotional state, making it difficult to provide a wide variety of recipe suggestions or services that respond to the user's emotions.In addition, in physical stores, there is a lack of services that suggest real-time recipes based on the food items that the user plans to purchase, or that encourage the purchase of necessary food and equipment, making it difficult to improve user satisfaction.

[1297] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating recipes based on the acquired food list and preference information, means for identifying and recognizing foods in the refrigerator through image analysis, means for referencing past usage data and personalizing recipes based on the user's preferences, means for automatically generating and sending a delivery request when necessary foods and utensils are missing, means for recognizing the user's emotional state and adjusting the recipe based on the emotional state, means for photographing foods to be purchased in the store and suggesting recipes, means for suggesting purchases when necessary foods and utensils are missing, and means for providing the user with recipes and cooking instructions. This enables a wide variety of recipes to be suggested based on the user's emotional state and the foods to be purchased, providing a comfortable shopping and cooking experience in a physical store.

[1298] "Acquired food list and preference information" refers to information indicating the food items currently owned by the user and the user's food preferences.

[1299] "Image analysis" is the technique of analyzing digital images to identify specific objects and extract information.

[1300] "Food in refrigerator" means food currently in a user's refrigerated storage unit.

[1301] "Past Usage Data" means historical information and related data generated by a User's use of the System.

[1302] "User Preferences" is information that indicates a user's personal preferences regarding foods and recipes that they like.

[1303] "Personalizing recipes" means adjusting recipes based on the user's individual preferences and past usage data, and suggesting recipes in the most optimal form for each individual user.

[1304] A "delivery request" is order information for delivery of necessary food and equipment to a specified location.

[1305] "User's emotional state" refers to the user's mental and emotional state as determined by analyzing voice and facial expression data.

[1306] "Foods to be purchased" refers to foods that a user intends to purchase in a physical store.

[1307] "Real-time recipe suggestions" means instantly generating and suggesting recipes based on the current situation and user input.

[1308] "Cooking instructions" are the detailed instructions and steps required to prepare a particular food product.

[1309] The system for realizing this invention is mainly composed of a server, a terminal, and a user. Each component and its operation will be explained below.

[1310] 1. Server Functions

[1311] The server has the following functions:

[1312] A means for generating recipes based on the obtained food list and preference information:

[1313] The server generates appropriate recipes based on the food list and cooking preferences entered or photographed by the user, and uses a generative AI model to suggest recipes that best match the ingredients and preferences.

[1314] Image analysis to identify and identify food items in refrigerators:

[1315] The system receives image data of the refrigerator sent by the user and uses an image analysis algorithm (e.g., Google Cloud Vision API) to identify the type and amount of food.

[1316] Ways to personalize recipes based on user preferences using past usage data:

[1317] The server uses past usage history data to suggest recipes tailored to the user's individual preferences, prioritizing recipes that have been popular in the past and ingredients that are frequently used.

[1318] A way to automatically generate and send delivery requests when needed food and equipment are in short supply:

[1319] If the server is running low on ingredients or cooking equipment required for the generated recipe, it will automatically send an order in conjunction with a delivery service and arrange for prompt delivery.

[1320] A way to recognize the user's emotional state and adjust recipes based on that emotional state:

[1321] The server analyzes voice and facial expression data to recognize the user's emotional state. For example, if the user is feeling stressed, it will prioritize recipes that are easy to prepare and have a relaxing effect.

[1322] A way to take a photo of the food you plan to buy in the store and get recipe suggestions:

[1323] The server takes a photo of the food the user plans to buy in a physical store, analyzes the image, and suggests suitable recipes, supporting the user's shopping and increasing their desire to buy.

[1324] Providing suggested purchases of necessary food and equipment if they are in short supply:

[1325] If the user is running low on food or equipment that he or she plans to purchase, the server will indicate this and suggest the purchase.

[1326] 2. Device Features

[1327] The device (e.g., smartphone, tablet) has the following functions:

[1328] Getting user input:

[1329] It offers a variety of input methods, including taking pictures of the refrigerator, entering a food list into a text box, and collecting voice and facial expression data.

[1330] Result display and cooking assistance:

[1331] The system supports cooking by displaying recipe data and cooking instructions sent from the server in an easy-to-understand manner to the user, and also provides relaxing music and video links as needed.

[1332] 3. User Behavior

[1333] Users use the system to perform the following actions:

[1334] Take and upload a picture of the food in your refrigerator.

[1335] Enter your ingredient list and cooking preferences into the text boxes.

[1336] Provides voice and facial expression data.

[1337] Order the necessary ingredients and equipment based on the proposed recipe.

[1338] Examples of specific examples and prompts

[1339] For example, if a user takes a photo of "tomatoes, cheese, pasta" in the refrigerator and the foods they have purchased, and provides the audio data to the app:

[1340] The server will parse the message assuming the following prompt:

[1341] Example prompt sentence:

[1342] "Generate the best recipe from the ingredients you have based on the user's emotional state. The current ingredients are tomatoes, cheese, and pasta."

[1343] This allows for flexible and personalized recipe suggestions tailored to the user's needs, providing a seamless shopping and cooking experience even in physical stores.

[1344] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1345] Step 1:

[1346] The user uses a smartphone or tablet to take pictures of the food in the refrigerator or the food they plan to buy. The user also inputs the food list into a text box and provides voice and facial expression data. The input data consists of image data, text data, and voice and facial expression data.

[1347] Step 2:

[1348] The device sends the collected image data, text data, and voice and facial expression data to the server, where it is entered as user input data.

[1349] Step 3:

[1350] The server uses image analysis algorithms to identify and identify the food in the refrigerator or the food you plan to buy from the image data sent to it. The extracted food data is output. At this stage, specific food names such as "tomato," "cheese," and "pasta" are obtained from the image data.

[1351] Step 4:

[1352] The server analyzes the voice and facial expression data to recognize the user's emotional state. It uses an emotion recognition engine (e.g., Google Cloud Speech-to-Text) to perform the analysis and outputs the user's emotional state. For example, it determines whether the user is relaxed or stressed.

[1353] Step 5:

[1354] The server uses a generative AI model based on the acquired food list and preference information to generate recipes. The input is food data and emotional state data, and the output is a suggested recipe. At this stage, a specific recipe such as "pasta with tomato and cheese" is output.

[1355] Step 6:

[1356] The server references past usage data and personalizes recipes based on the user's preferences. Entering past usage data results in personalized recipes. For example, if a user has previously preferred "tomato pasta," the recipe will be adjusted to take this into account.

[1357] Step 7:

[1358] The server automatically generates and sends a delivery request if the food and equipment required for the generated recipe are missing. The input is the missing food and equipment data, and the output is the generated delivery request. For example, if "cheese" is missing, a request is generated suggesting the purchase of that item.

[1359] Step 8:

[1360] The server sends the final recipe data, cooking instructions, and information on missing foods and equipment to the device. The input is the generated recipe and cooking instructions data, and the output is the information sent to the device. Specifically, the step-by-step cooking instructions and instructions for purchasing "cheese" are displayed.

[1361] Step 9:

[1362] The device finally displays the recipe and cooking instructions to the user, and provides relaxing music and video links as needed.The user can then cook based on the displayed information and enjoy the suggested recipe.

[1363] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1364] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1365] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1366] [Fourth embodiment]

[1367] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1368] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1369] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1370] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1371] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1372] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1373] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1374] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1375] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1376] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1377] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1378] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1379] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1380] The present invention provides a system for solving the problems of cooking in a rut and food waste in busy daily lives. Below, an embodiment of this system is described.

[1381] 1. Getting User Input

[1382] To help users efficiently utilize the ingredients in their refrigerator and enjoy a wide variety of dishes, the device first acquires input data. The device provides users with the options to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their favorite dishes. This allows the device to collect information about the user.

[1383] 2. Image Analysis and Data Conversion

[1384] The image data of the refrigerator sent by the device is received by the server. The server uses an image analysis algorithm to identify and identify the ingredients present in the refrigerator. At the same time, the list of ingredients and cooking preferences are also analyzed and converted into text data. For example, ingredients such as "carrots," "bell peppers," "chicken," and "eggs" can be recognized from the image and recorded as text data.

[1385] 3. Recipe Creation and Search

[1386] The server generates recipes based on the acquired ingredient list and preference information. Using a generative AI model, it suggests recipes that make the most of the ingredients the user has. For example, if a user has chicken, eggs, carrots, and green peppers, it will suggest recipes such as oyakodon (chicken and egg rice bowl) or stir-fry. It also references an existing recipe database to search for and provide related recipe information.

[1387] 4. Personalizing your data

[1388] The server refers to the user's past usage data and personalizes recipes based on the user's preferences and habits. It also takes into account the expiration dates of ingredients and makes suggestions to minimize food waste. For example, it can prevent food waste by suggesting recipes that prioritize ingredients with a short expiration date.

[1389] 5. Delivery of missing ingredients and equipment

[1390] If the server does not have the ingredients or equipment required for the generated recipe on hand, it automatically generates a delivery request and sends an order to an external supply service. The delivery service quickly arranges for the missing ingredients and equipment and delivers them to the user. For example, if there is a shortage of "chicken," that information is sent to the delivery service and arrangements are made to procure it.

[1391] 6. Results display and cooking assistance

[1392] The server sends the final recipe data, cooking instructions, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner to the user, providing text and visual content to guide the user through the cooking methods and procedures. For example, the device displays detailed instructions, cooking time, and a list of necessary equipment for making "Oyakodon."

[1393] Specific examples

[1394] Example 1: Upload an image of a refrigerator

[1395] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app.

[1396] 2. The server analyzes the image and recognizes "carrots," "bell peppers," "chicken," and "eggs."

[1397] 3. The server generates a recipe for "Oyakodon" based on these ingredients and sends the cooking instructions to the terminal.

[1398] 4. The device displays cooking instructions to the user.

[1399] Example 2: Enter a list of ingredients

[1400] 1. A user types "tomato, cheese, pasta" into a text box.

[1401] 2. The server analyzes this, searches for related recipes, and suggests "pasta with tomatoes and cheese."

[1402] 3. The server generates a recipe that takes expiration dates into account and sends the details to the device.

[1403] 4. The device displays the recipe and cooking instructions to the user.

[1404] In this way, the present invention can effectively utilize leftover ingredients in the refrigerator, preventing daily cooking from becoming monotonous and providing users with a fresh and healthy diet.

[1405] The processing flow will be explained below.

[1406] Step 1:

[1407] A user launches the application using a device such as a smartphone or tablet, which then provides the user with the option to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their preferred dish.

[1408] Step 2:

[1409] Users can take a picture of their refrigerator and upload it to the application, or enter their ingredient list and cooking preferences into a text box.

[1410] Step 3:

[1411] The terminal sends the user's input data (image data or text data) to the server.

[1412] Step 4:

[1413] The server uses an image analysis algorithm to analyze the image data received from the refrigerator. Specifically, it applies a food recognition model to identify ingredients from the image. This allows ingredients such as "carrots," "bell peppers," "chicken," and "eggs" to be identified and classified.

[1414] Step 5:

[1415] The server performs text analysis on the user's input of ingredients and cooking preferences to extract ingredients and cooking genres, which are also used to generate recipes.

[1416] Step 6:

[1417] The server uses the generative AI model to generate customized recipes based on the user's food list and preferences, and also references existing recipe databases such as Classy to search and retrieve related recipe information.

[1418] Step 7:

[1419] The server references the user's past usage data, taking into account information such as past favorite recipes and commonly used ingredients, and personalizes recipes based on the user's preferences.

[1420] Step 8:

[1421] The server creates a list of ingredients and equipment needed for the generated recipe, checking the user's ingredient list to ensure all ingredients are available.

[1422] Step 9:

[1423] The server adjusts recipes taking into account the expiration dates of ingredients, suggesting recipes that prioritize ingredients with close expiration dates and preventing food waste.

[1424] Step 10:

[1425] If the server is running low on ingredients or equipment, it automatically generates a delivery request, which is then sent to an external supply service (e.g., an online shopping site).

[1426] Step 11:

[1427] The server receives confirmation from the delivery service and sends a notification to the user, for example, "Your chicken will arrive in 30 minutes."

[1428] Step 12:

[1429] The server sends the final recipe data, cooking instructions, and information on missing ingredients and utensils to the terminal.

[1430] Step 13:

[1431] The device receives information and displays it to the user in an easy-to-understand manner, such as providing recipe steps, necessary equipment, and links to cooking videos.

[1432] Step 14:

[1433] The user begins cooking according to the recipe displayed on the device, with the help of a timer and additional tips provided by the app.

[1434] Step 15:

[1435] The device records the user's cooking progress, sends the data back to the server, and stores it in a database for future personalization.

[1436] Example 1

[1437] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1438] In today's busy lifestyles, cooking at home can easily become monotonous, and food in the refrigerator is often underutilized and thrown away. While this problem differs from household to household, the following issues need to be resolved: First, to efficiently utilize the food in the refrigerator and provide nutritionally balanced meals. Second, to suggest personalized recipes that take into account the user's preferences and past cooking history. Third, to assist the user in cooking by quickly arranging for ingredients and utensils that are in short supply.

[1439] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1440] In this invention, the server includes means for generating recipes based on the acquired ingredient list and preference information, means for identifying and identifying ingredients in the refrigerator through image analysis, means for referencing past usage data and personalizing recipes based on the user's preferences, means for automatically generating and sending a delivery request when necessary ingredients and cooking utensils are in short supply, means for providing the user with recipes and cooking instructions, means for generating recipes based on input data using a generative AI model, and means for automatically generating prompt statements, inputting them into the AI ​​model, and generating recipes based on the results. This makes it possible to efficiently use ingredients in the refrigerator, provide personalized recipes that suit the user's preferences, and minimize food waste.

[1441] The "obtained ingredient list" refers to the information about ingredients in the refrigerator entered by the user, including image analysis results and text input data.

[1442] "Preference information" is data about a user's preferences based on their cooking preferences, past usage history, etc.

[1443] The "means for generating recipes" is a function that creates new recipes using a generative AI model and recipe database based on the acquired ingredient list and preference information.

[1444] "Image analysis" is a technique that processes image data to identify ingredients in the refrigerator. It includes image recognition algorithms and machine learning models.

[1445] "Means for identification" refers to the function of identifying and specifying ingredients based on certain criteria from the information obtained by image analysis.

[1446] "Past Usage Data" refers to all data from a user's previous use of the system, including information about recipes selected and ingredients used.

[1447] "Personalization" refers to a function that suggests individually appropriate recipes based on the user's past usage data and preferences.

[1448] "Delivery Request" refers to a request to send an order to an external supply service when necessary ingredients or cooking equipment are in short supply.

[1449] "Means for providing cooking instructions" is a function that clearly provides specific steps and information on necessary equipment when a user cooks a dish.

[1450] A "generative AI model" is an artificial intelligence model used for complex data processing such as image analysis and recipe generation, as well as natural language processing.

[1451] A "prompt sentence" is a text sentence that expresses a question or request that is input to a generative AI model, and the AI ​​generates a response based on this.

[1452] The present invention is a system that solves the problems of cooking in a rut and food waste in busy daily lives. This system efficiently utilizes ingredients in the refrigerator and suggests a wide variety of dishes, providing users with a fresh and healthy diet. Detailed embodiments for implementing this system are described below.

[1453] Getting User Input

[1454] Users use a dedicated application on their smartphone or PC to input information about the ingredients in their refrigerator. The input method is as follows:

[1455] 1. Take a picture of the ingredients in your refrigerator and upload it.

[1456] 2. Enter the ingredients list in the text box.

[1457] 3. Use the drop-down menu to select your preferred cuisine.

[1458] Image analysis and data conversion

[1459] The device sends image data, text data, or data on the user's favorite dishes to the server. The server then uses an image analysis algorithm (using, for example, OpenCV or TensorFlow) to identify the ingredients in the refrigerator. For example, it identifies "carrots," "bell peppers," "chicken," and "eggs" from the image and saves them as text data. This allows the server to obtain a list of ingredients and the user's favorite dishes.

[1460] Recipe Generation

[1461] The server generates a recipe using a generative AI model (for example, GPT-3 or a similar natural language processing model) based on the acquired ingredient list and preference information. Specifically, the server automatically generates a prompt sentence and inputs it into the generative AI model. For example, a prompt sentence such as "What recipes use chicken, eggs, carrots, and green peppers?" is generated and input into the AI ​​model. The recipe is generated based on the results obtained from the AI ​​model.

[1462] Data Personalization

[1463] The server refers to the user's past usage data and proposes personalized recipes that take into account the user's preferences and the expiration dates of ingredients, thereby prioritizing the use of ingredients with an approaching expiration date, thereby minimizing food waste.

[1464] Delivery of missing ingredients and cooking equipment

[1465] If the server is running low on ingredients or cooking utensils required for the generated recipe, it automatically generates a delivery request and sends it to an external supply service. For example, if there is a shortage of chicken, the server sends that information to the delivery service and quickly arranges for the missing ingredients or cooking utensils.

[1466] Result display and cooking assistance

[1467] The server sends the final recipe data, cooking instructions, and information on missing ingredients and cooking utensils to the device. The device then displays this information in an easy-to-understand manner to the user, guiding them through the cooking methods and procedures. For example, the device displays detailed instructions for making an "oyakodon," cooking time, and a list of necessary utensils. It also displays text and visual content (images and videos) to guide the user through the cooking process.

[1468] Specific examples

[1469] Example 1: Upload an image of a refrigerator

[1470] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app.

[1471] 2. The server analyzes the image and identifies "carrots," "bell peppers," "chicken," and "eggs."

[1472] 3. The server generates a recipe for "Oyakodon" based on these ingredients and sends the cooking instructions to the terminal.

[1473] 4. The device displays cooking instructions to the user.

[1474] Example 2: Enter a list of ingredients

[1475] 1. A user types "tomato, cheese, pasta" into a text box.

[1476] 2. The server analyzes this, searches for related recipes, and suggests "pasta with tomatoes and cheese."

[1477] 3. The server generates a recipe that takes expiration dates into account and sends the details to the device.

[1478] 4. The device displays the recipe and cooking instructions to the user.

[1479] In this way, users can efficiently utilize the ingredients in their refrigerator and enjoy fresh and healthy meals without getting bored with their daily cooking routine.

[1480] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1481] Step 1: Getting User Input

[1482] Users launch the application on their smartphone or PC and enter information about the ingredients in their refrigerator by taking a picture of the refrigerator and uploading it, entering the list of ingredients in a text box, or selecting the option to choose their favorite dish.

[1483] Input: Image data of the contents of the refrigerator, text data, or favorite food information

[1484] Output: User input data (image data, text data, favorite food information)

[1485] Step 2: Image analysis and data conversion

[1486] The terminal transmits the acquired user input data to the server.

[1487] The server processes the received image data using an image analysis algorithm (e.g., OpenCV or TensorFlow) to identify the ingredients in the refrigerator. For example, it identifies "carrots," "bell peppers," "chicken," and "eggs" from the image.

[1488] Input: User-entered data (image data, text data, favorite food information)

[1489] Data processing: Identification and classification of ingredients using image analysis

[1490] Output: Identified ingredients list (text data)

[1491] Step 3: Generate the recipe

[1492] Based on the ingredient list and preference information acquired by the server, a recipe is generated using a generative AI model (e.g., GPT-3). The server generates a prompt sentence and inputs it into the generative AI model. For example, a prompt sentence such as "What is a recipe that uses chicken, eggs, carrots, and green peppers?" is generated and input into the AI ​​model.

[1493] Input: Ingredient list, preference information

[1494] Data processing: Recipe generation using generative AI models

[1495] Output: Suggested recipe (text data)

[1496] Step 4: Personalize your data

[1497] The server references the user's past usage data and generates personalized recipes that take into account the user's preferences and the expiration dates of ingredients. Specifically, it suggests recipes that prioritize ingredients with an approaching expiration date.

[1498] Input: Ingredient list, preference information, past usage data

[1499] Data processing: Consideration of preference analysis and expiration date data

[1500] Output: Personalized recipe (text data)

[1501] Step 5: Delivery of missing ingredients and cooking equipment

[1502] The server creates a list of ingredients and cooking tools needed based on the recipe. If any of the ingredients are missing, it automatically generates a delivery request and sends it to an external supply service. For example, if chicken is in short supply, it requests a delivery service to procure it.

[1503] Input: Personalized recipe, information about ingredients and cooking equipment the user has on hand

[1504] Data processing: Identifying missing ingredients and cooking utensils, generating automatic delivery requests

[1505] Output: Delivery request

[1506] Step 6: Displaying results and cooking assistance

[1507] The server sends the final recipe data, along with information on missing ingredients and cooking utensils, to the device. The device then displays this information in an easy-to-understand manner to the user, guiding them through the cooking method and steps. For example, it displays detailed instructions, cooking time, and a list of necessary utensils for making "Oyakodon."

[1508] Input: Final recipe data, cooking instructions, missing ingredients and utensils information

[1509] Data processing: Format conversion of cooking instructions and necessary information

[1510] Output: Cooking guide display for the user (text and visual content)

[1511] Through these steps, the system can efficiently utilize the ingredients in the refrigerator and provide users with a fresh and healthy diet.

[1512] (Application example 1)

[1513] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1514] Conventional food management systems do not adequately suggest recipes that efficiently utilize ingredients in the refrigerator, resulting in food waste. Furthermore, recipe suggestions that take into account the user's preferences and past cooking history are limited. In particular, the lack of a convenient way to provide cooking instructions or delivery of missing ingredients in busy daily lives has been a major problem for users.

[1515] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1516] In this invention, the server includes: means for generating recipes based on the acquired food list and preference information; means for identifying and recognizing foods in the refrigerator through image analysis; means for referencing past usage data and personalizing recipes based on the user's preferences; means for automatically generating and sending delivery requests when necessary foods and utensils are missing; means for providing recipes and cooking instructions to the user; means for processing food images taken with a smartphone; means for automatically recognizing the acquired food list using an image analysis algorithm; means for using a generative AI model to generate and suggest optimal recipes based on the acquired food list; and means for transmitting the missing list to a delivery service. This enables effective use of ingredients in the refrigerator, reduction of food waste, personalized recipe suggestions for users, and easy delivery ordering.

[1517] - "Food List" refers to a list of ingredients owned by the user.

[1518] "Preference information" refers to information about the cuisine and taste preferences of a user that has been selected in the past.

[1519] The "means for generating recipes" is a function that automatically creates recipes showing cooking methods and ingredients based on the acquired food list and preference information.

[1520] "Image analysis" refers to techniques and processes that process digital image data to extract specific information.

[1521] "Means for identifying and discerning food in a refrigerator" refers to a technology that uses image analysis to recognize and identify food items present in a refrigerator.

[1522] "Past usage data" refers to historical information such as ingredients used by the user, recipes selected, and ingredients ordered.

[1523] "Means for personalizing recipes based on user preferences" refers to technology that references past usage data and suggests recipes tailored to the user's specific preferences.

[1524] "Means for automatically generating and sending delivery requests" refers to a function that automatically detects when necessary food or equipment is missing and sends an order to an external delivery service.

[1525] "Means for providing recipes and cooking instructions" refers to a function that displays or communicates suggested recipes and their cooking instructions to the user.

[1526] The "means for processing food images taken by a smartphone" refers to the process of receiving image data taken by a user, analyzing it, and extracting a list of foods to be used.

[1527] An "image analysis algorithm" is a set of mathematical models or programs that process image data to recognize specific objects or text information.

[1528] A "generative AI model" is an artificial intelligence model that uses machine learning to generate new recipes and information from data.

[1529] A "delivery service" is a service organization that delivers ingredients and equipment to users.

[1530] The present invention is a system that efficiently utilizes ingredients in a refrigerator, provides a wide variety of dishes, and prevents food waste. This system starts when a user takes a photo of the ingredients in the refrigerator with their smartphone and uploads it to an application.

[1531] Hardware and software used

[1532] Hardware:

[1533] Smartphone: Taking pictures of ingredients and using applications.

[1534] Server: Image analysis, recipe generation, and delivery request processing.

[1535] Delivery service computer system: Arranges delivery of ingredients and cooking equipment.

[1536] software:

[1537] Flask: A web application framework in Python.

[1538] PIL (Pillow): Image processing library.

[1539] food_recognition: Food recognition algorithm.

[1540] recipe_generation: Recipe generation algorithm.

[1541] delivery_service: Third-party delivery service API.

[1542] Processing flow

[1543] 1. Getting User Input

[1544] Users take a photo of the ingredients in their refrigerator with their smartphone and upload the image to the application.

[1545] Alternatively, you're given the option to manually enter the ingredient list into a text box.

[1546] Information about the user's preferences and past cooking tastes can also be entered.

[1547] 2. Image Analysis and Data Conversion

[1548] The server receives the image data sent from the smartphone and reads the image data using Pillow.

[1549] Ingredients are identified using an image analysis algorithm (food_recognition), and the information is converted into text data.

[1550] For example, "tomato," "cheese," and "beef" can be identified from an image of a refrigerator and recorded as text data.

[1551] 3. Recipe Generation

[1552] A generative AI model (recipe_generation) is used to generate recipes based on the acquired ingredient list and user preference information.

[1553] For example, we suggest a recipe for Bolognese pasta using tomatoes, cheese, and beef.

[1554] 4. Creating a delivery request

[1555] The server checks the ingredients and equipment required for the generated recipe, and if any are missing, it automatically generates a delivery request for them.

[1556] It calls the delivery service's API and arranges for the necessary ingredients and equipment to be delivered to the user.

[1557] 5. Results display and cooking assistance

[1558] The server sends the final recipe data, cooking instructions, and information on missing ingredients and utensils to the smartphone.

[1559] The smartphone displays this information in an easy-to-understand way for the user, guiding them through the cooking process with visual and textual content.

[1560] Specific examples

[1561] Example 1: Uploading food images

[1562] Users take a picture of the inside of their refrigerator with their smartphone and upload it to the app.

[1563] The server analyzes the image and recognizes "carrots," "bell peppers," "chicken," and "eggs."

[1564] The server uses a generative AI model to generate a recipe for "Oyakodon" based on these ingredients and sends the cooking instructions to a smartphone.

[1565] The smartphone displays cooking instructions to the user.

[1566] Example 2: Ordering delivery of ingredients that are in short supply

[1567] A user uploads a picture of their refrigerator and discovers that they are out of "chicken."

[1568] The server generates a delivery request and places an order for "chicken" through the delivery service API.

[1569] The delivery service arranges for the chicken to be delivered to the user.

[1570] Example 3: Example of a prompt statement

[1571] Prompt: "How can I build a system that allows you to upload a picture of the ingredients in your refrigerator, suggest recipes based on the images, and order delivery for any missing ingredients? What are the basic features this system should have?"

[1572] As a result, this system can make effective use of ingredients in the refrigerator, reduce food waste, and provide users with personalized recipe suggestions and delivery services.

[1573] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1574] Step 1:

[1575] The user takes a photo of the food in the refrigerator with their smartphone and uploads the image to the application.

[1576] Specific actions

[1577] Users use their smartphone camera to take a picture of the ingredients in their refrigerator and upload the image to a dedicated app, which also offers the option to input user preference information and allows users to provide past usage data.

[1578] Input and Output

[1579] Input: Image data of ingredients, preference information

[1580] Output: Image data and preference information sent to the server

[1581] Step 2:

[1582] The server receives the image data and loads the image using Pillow.

[1583] Specific actions

[1584] The server receives the uploaded image data and reads the image using the Pillow image processing library, thereby obtaining the digital image data.

[1585] Input and Output

[1586] Input: Image data sent from a smartphone

[1587] Output: Loaded image data

[1588] Step 3:

[1589] The server uses an image analysis algorithm (food_recognition) to identify ingredients and convert them into text data.

[1590] Specific actions

[1591] The server analyzes the received image data using the food_recognition algorithm, identifies the ingredients in the image, converts the identified ingredients into text data, and creates a list.

[1592] Input and Output

[1593] Input: Image data

[1594] Output: Ingredient list (text data)

[1595] Step 4:

[1596] The server uses a generative AI model (recipe_generation) to generate a recipe based on the ingredient list and preference information.

[1597] Specific actions

[1598] The server uses a generative AI model to create an appropriate recipe based on the acquired ingredient list and the user's preferences. The generated recipe also includes specific cooking steps.

[1599] Input and Output

[1600] Input: Ingredient list, preference information

[1601] Output: Generated recipe and cooking instructions

[1602] Step 5:

[1603] The server checks for missing ingredients and equipment based on the generated recipe and generates a delivery request.

[1604] Specific actions

[1605] The server compares all ingredients included in the generated recipe with the user's current inventory, identifies any missing ingredients or utensils, and generates and sends a request to an external delivery service API.

[1606] Input and Output

[1607] Input: Generated recipe, ingredient list

[1608] Output: Delivery request

[1609] Step 6:

[1610] The server sends the final recipe data, cooking instructions, and delivery request results to the smartphone.

[1611] Specific actions

[1612] The server then sends the generated recipe data, cooking instructions, and delivery results for missing ingredients and utensils to a smartphone, where users can check the information using the smartphone app.

[1613] Input and Output

[1614] Input: Final recipe data, cooking instructions, delivery request results

[1615] Output: Recipe data and cooking instructions displayed on a smartphone

[1616] Through these steps, the system can effectively utilize the ingredients in the refrigerator and provide personalized recipe suggestions and delivery services.

[1617] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1618] The present invention is a system that efficiently utilizes ingredients and utensils that a user has, and also suggests recipes and cooking methods according to the user's emotional state. The following describes an embodiment of this system.

[1619] 1. Getting User Input

[1620] To help users efficiently utilize the ingredients in their refrigerator and enjoy a wide variety of dishes, the device first acquires input data. The device provides users with the option to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their preferred dishes. Additionally, the device also has the ability to collect voice and facial expression data to recognize the user's emotions.

[1621] 2. Image Analysis and Data Conversion

[1622] The image data of the refrigerator sent by the device is received by the server. The server uses an image analysis algorithm to identify and identify the ingredients present in the refrigerator. At the same time, the list of ingredients and cooking preferences are also analyzed and converted into text data. For example, ingredients such as "carrots," "bell peppers," "chicken," and "eggs" can be recognized from the image and recorded as text data.

[1623] 3. Emotional Recognition

[1624] The device collects the user's voice and facial expression data and sends it to the server. The server then uses an emotion engine to analyze and recognize the user's emotions from this data. For example, it analyzes changes in voice tone and facial expressions to determine whether the user is relaxed or stressed.

[1625] 4. Recipe Creation and Search

[1626] The server generates recipes based on the acquired ingredient list and preference information. Using a generative AI model, it suggests recipes that make the most of the ingredients the user has. It also references an existing recipe database to search for and provide related recipe information. For example, if a user has chicken, eggs, carrots, and green peppers, it will suggest recipes such as oyakodon (chicken and egg rice bowl) or stir-fry. Depending on the user's emotional state, it also prioritizes suggestions of very simple recipes and dishes with a relaxing effect.

[1627] 5. Personalization of Data

[1628] The server refers to the user's past usage data and personalizes recipes based on the user's preferences and habits. It also takes into account the expiration dates of ingredients and makes suggestions to minimize food waste. For example, it can prevent food waste by suggesting recipes that prioritize ingredients with a short expiration date.

[1629] 6. Delivery of missing ingredients and equipment

[1630] If the server does not have the ingredients or equipment required for the generated recipe on hand, it automatically generates a delivery request and sends an order to an external supply service. The delivery service quickly arranges for the missing ingredients and equipment and delivers them to the user. For example, if there is a shortage of "chicken," that information is sent to the delivery service and arrangements are made to procure it.

[1631] 7. Results display and cooking assistance

[1632] The server sends the final recipe data, cooking instructions, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner to the user, providing text and visual content to guide the user through the cooking methods and procedures. For example, the device may display detailed instructions, cooking time, and a list of necessary equipment for making "Oyakodon." Furthermore, the device may also provide relaxing music and video links depending on the user's emotional state.

[1633] Specific examples

[1634] Example 1: Uploading images and audio data of a refrigerator

[1635] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app, which also collects audio data.

[1636] 2. The server analyzes the image and recognizes "carrot," "bell pepper," "chicken," and "egg." At the same time, it analyzes the voice data and recognizes the user's emotional state.

[1637] 3. The server generates a recipe for "Oyakodon" based on this information and sends the cooking instructions to the device. If the user is feeling stressed, it also suggests relaxing music.

[1638] 4. The device displays and plays cooking instructions and relaxing music to the user.

[1639] Example 2: Input ingredients list and facial expression data

[1640] 1. The user enters "tomato, cheese, pasta" into the text box, and facial expression data is collected using the camera.

[1641] 2. The server analyzes this, searches for related recipes, and suggests "Tomato and Cheese Pasta." It also determines from facial expression data that the user is relaxed.

[1642] 3. The server generates a recipe that takes expiration dates into account and sends the details to the device.

[1643] 4. The device displays the recipe and cooking instructions to the user.

[1644] In this way, the present invention effectively utilizes leftover ingredients in the refrigerator and suggests recipes that take into consideration the user's emotional state, preventing cooking from becoming monotonous, reducing food waste, and providing a richer diet.

[1645] The processing flow will be explained below.

[1646] Step 1:

[1647] A user launches the application using a device such as a smartphone or tablet. The device provides the user with the options to take and upload a picture of their refrigerator, enter a list of ingredients in a text box, and select their preferred dish. It also prompts them to collect voice data and capture facial expression data.

[1648] Step 2:

[1649] Users take a picture of their refrigerator and upload it to the application, and are also asked to record a voice message or enter facial expression data via the camera to ascertain its emotional state.

[1650] Step 3:

[1651] The terminal sends the user's input data (image data, voice data, facial expression data, or text data) to the server.

[1652] Step 4:

[1653] The server analyzes the image data of the refrigerator it receives using an image analysis algorithm. Specifically, it uses a food recognition model to identify ingredients from the image. As a result, ingredients such as "carrots," "bell peppers," "chicken," and "eggs" in the refrigerator are identified.

[1654] Step 5:

[1655] The server analyzes the user's ingredient list and cooking preferences received as text data and extracts the ingredient list and cooking categories.

[1656] Step 6:

[1657] The server uses an emotion engine to analyze the transmitted voice and facial expression data and recognize the user's emotional state. For example, it can determine whether the user is relaxed or stressed based on changes in voice tone and facial expression.

[1658] Step 7:

[1659] The server generates recipes based on the acquired ingredient list and the user's recognized emotional state. Using a generative AI model, the server suggests recipes that make the most of the ingredients the user has. The server then adjusts the recipes according to the user's emotional state. For example, if the user is tired, the server suggests simple recipes that have a relaxing effect.

[1660] Step 8:

[1661] The server uses past usage data to personalize recipes based on the user's preferences and habits, and also takes into account expiration dates of ingredients, prioritizing recipes that minimize food waste.

[1662] Step 9:

[1663] The server creates a list of ingredients and tools required for the generated recipe, referencing the user's ingredient list to see what is available and what is missing.

[1664] Step 10:

[1665] If the server is missing ingredients or equipment, it automatically generates a delivery request, which is then sent to an external supply service.

[1666] Step 11:

[1667] The server receives confirmation from the delivery service and notifies the user when the missing ingredients or equipment will arrive, for example, "The chicken will arrive in 30 minutes."

[1668] Step 12:

[1669] The server sends the final recipe data, cooking instructions, and information on missing ingredients and utensils to the terminal.

[1670] Step 13:

[1671] The device displays the information received in an easy-to-understand manner to the user, providing recipe steps, necessary utensils, and links to cooking videos through text and visual content. It also displays relaxing music and videos based on the user's emotional state.

[1672] Step 14:

[1673] The user begins cooking by following the recipe displayed on the device, with the help of a timer and additional tips provided by the app.

[1674] Step 15:

[1675] The device records the user's cooking progress, sends the data back to the server, and stores it in a database for future personalization.

[1676] Example 2

[1677] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1678] Currently, many households are unable to efficiently utilize the ingredients in their refrigerators, resulting in food waste. It is also difficult for systems to suggest dishes that reflect the user's emotional state, which can reduce meal satisfaction. Furthermore, if the necessary ingredients or cooking utensils are in short supply, it takes time to procure them, making it difficult to start cooking quickly. There is a need to solve these issues and provide users with an efficient and satisfying cooking experience.

[1679] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1680] In this invention, the server includes means for receiving and analyzing user input as text data and image data, means for identifying and identifying foods in the refrigerator through image analysis, means for generating recipes based on the acquired food list and preference information, means for analyzing the user's emotional state using voice and facial expression data, means for suggesting recipes according to the user's emotional state, means for personalizing recipes based on the user's preferences by referencing past usage data, means for automatically generating and sending a delivery request when necessary foods and utensils are in short supply, means for providing the user with recipes and cooking instructions, and means for generating recipes using a generative AI model. This makes it possible to efficiently use ingredients in the refrigerator, suggest recipes according to the user's emotional state, and quickly procure the necessary ingredients and utensils.

[1681] "User input" refers to text data, image data, voice data, and facial expression data that a user provides to the system.

[1682] "Text data" refers to data entered by the user in the form of text, such as a list of ingredients or preference information.

[1683] "Image data" refers to images of food and other visual information in the refrigerator taken with a camera and uploaded to the system.

[1684] "Audio Data" refers to audio information collected for the purpose of analyzing a user's emotional state.

[1685] "Facial expression data" refers to emotional information extracted from images of a user's facial expressions captured by a camera.

[1686] "Image analysis" refers to the process of analyzing transmitted image data to identify and identify the food items in the refrigerator.

[1687] A "food list" refers to data that records in text format all the food in your refrigerator.

[1688] "Preference information" refers to information that represents a user's preferences for food and cooking.

[1689] "Emotional state" refers to the user's current psychological and emotional state, analyzed from their voice and facial expression data.

[1690] A "recipe" is a document or piece of information that lists the ingredients and steps needed to prepare a particular dish.

[1691] "Generative AI model" refers to an algorithmic model that uses artificial intelligence to generate new recipes.

[1692] "Past usage data" refers to data that records a user's system usage history.

[1693] "Food waste" refers to food that is discarded without being consumed.

[1694] "Delivery Request" means an order request sent to an external supply service to procure needed food and / or equipment.

[1695] "Personalization" refers to the process of providing users with optimized suggestions and services based on their past usage history and preferences.

[1696] The present invention is a system that efficiently utilizes the ingredients and cooking utensils that a user has, and also suggests recipes and cooking methods that correspond to the user's emotional state. Specific implementation methods of this system are described in detail below.

[1697] Getting User Input

[1698] Users input ingredient information and their favorite dishes using a smartphone or PC. The device provides users with the ability to take and upload images of their refrigerator, a text box for entering a list of ingredients, and check boxes and drop-down menus for selecting their favorite dishes. The device also collects user emotional data through voice input and facial recognition using a camera.

[1699] Image analysis and data conversion

[1700] The server receives the image data of the refrigerator sent by the device and uses image analysis algorithms (e.g., OpenCV or TensorFlow) to identify and distinguish the ingredients in the refrigerator. For example, it recognizes "carrots," "bell peppers," "chicken," and "eggs" from the image of the refrigerator and records them as text data. In addition, it analyzes the text list of ingredients and cooking preferences entered by the user and stores them in a database.

[1701] Emotion recognition

[1702] The device sends the collected voice and facial expression data to a server, which then analyzes it using an emotion analysis engine (e.g., IBM Watson or Microsoft Azure's emotion analysis API). This determines whether the user is relaxed or stressed. The analysis results are then recorded in a database.

[1703] Recipe creation and search

[1704] The server uses a generative AI model (e.g., GPT-4) to generate new recipes based on the acquired ingredient list and emotional data. For example, if a user has chicken, eggs, carrots, and green peppers, it will suggest recipes such as oyakodon (chicken and egg rice bowl) or stir-fry. It will also search an existing recipe database to provide related recipe information. If the user is feeling stressed, it will prioritize suggestions of easy recipes and dishes that have a relaxing effect.

[1705] Data Personalization

[1706] The server refers to the user's past usage data and personalizes recipes based on the user's preferences. For example, if there is an ingredient that is close to its expiration date, it can suggest a recipe that prioritizes using that ingredient, thereby reducing food waste.

[1707] Delivery of missing ingredients and equipment

[1708] The server checks whether the user has the ingredients and equipment required for the generated recipe. If they are in short supply, it automatically generates a delivery request and sends an order to an external supply service. For example, if there is a shortage of "chicken," that information is sent to the delivery service and arrangements are made for timely procurement.

[1709] Result display and cooking assistance

[1710] The server sends the final recipe data, cooking instructions, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner to the user, providing text and visual content to guide the user through the cooking method and steps. For example, it may display detailed instructions, cooking time, and a list of necessary equipment for making "Oyakodon." It may also provide relaxing music or video links depending on the user's emotional state.

[1711] Specific examples

[1712] Example 1: Uploading images and audio data of a refrigerator

[1713] 1. The user takes a picture of the inside of the refrigerator with their smartphone and uploads it to the app, which also collects audio data.

[1714] 2. The device sends image and audio data to the server.

[1715] 3. The server analyzes the image and recognizes "carrot," "bell pepper," "chicken," and "egg." It also analyzes the voice data to understand the user's emotional state.

[1716] 4. The server generates a recipe for "Oyakodon" based on the recognized ingredients and emotional data. If it determines that the user is feeling stressed, it selects music with a relaxing effect.

[1717] 5. The server sends the generated recipe and music information to the device.

[1718] 6. The device displays and plays cooking instructions and relaxing music to the user.

[1719] Example 2: Input ingredients list and facial expression data

[1720] 1. The user enters "tomato, cheese, pasta" into the text box, and facial expression data is collected using the camera.

[1721] 2. The device sends the text and facial expression data to the server.

[1722] 3. The server analyzes the text data to search for relevant recipes and determines whether the user is relaxed based on facial expression data.

[1723] 4. The server generates a recipe for "Tomato and Cheese Pasta" and creates a recipe that takes into account the expiration date.

[1724] 5. The server sends the generated recipe to the device.

[1725] 6. The device displays the recipe and cooking instructions to the user.

[1726] In this way, the present invention effectively utilizes leftover ingredients in the refrigerator and suggests recipes that take into consideration the user's emotional state, preventing cooking from becoming monotonous, reducing food waste, and providing a richer diet.

[1727] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1728] Step 1: Getting User Input

[1729] Users open the app on their smartphone or PC and enter information about ingredients in their refrigerator and their favorite dishes, or take and upload images. Voice input and facial expression data from the camera are also collected. Image data, text data, voice data, and facial expression data are obtained as input, and this input data is sent to the system. The device collects this data, converts it into an appropriate format, and sends it to the server.

[1730] Step 2: Image analysis and data conversion

[1731] The server receives the image data of the refrigerator sent from the device and uses image analysis algorithms (e.g., OpenCV or TensorFlow) to identify the ingredients in the refrigerator. In this process, image data is input and converted into text data. For example, it identifies "carrots," "bell peppers," "chicken," and "eggs" from the image of the refrigerator and outputs them as text data. The server also analyzes the text list of ingredients and cooking preferences entered by the user and stores them in a database.

[1732] Step 3: Recognize emotions

[1733] The server receives the voice and facial expression data sent from the device and analyzes the user's emotional state using an emotion analysis engine (e.g., IBM Watson or Microsoft Azure's emotion analysis API). Voice and facial expression data are received as input, and they are analyzed to output information about the user's emotional state (relaxed, stressed, etc.). The analysis results are stored in a database.

[1734] Step 4: Creating and searching recipes

[1735] The server generates new recipes using a generative AI model (e.g., GPT-4) based on the acquired ingredient list, emotional data, and preference information. In this process, the ingredient list and emotional state are input as text data, and recipe suggestions based on them are output. For example, if a user has "chicken," "eggs," "carrots," and "green peppers," specific recipes such as "oyakodon" (chicken and egg rice bowl) or "stir-fry" are generated. The server also searches an existing recipe database and provides related recipe information. The preferred recipes change depending on the user's emotional state.

[1736] Step 5: Personalize your data

[1737] The server refers to the user's past usage data and personalizes recipes by taking into account the user's preferences, habits, and food expiration dates. The inputs are past usage history, preference information, and food expiration date data. Based on these, the server outputs a recipe optimized for the user. This recipe includes recipes that prioritize the use of ingredients with an approaching expiration date.

[1738] Step 6: Delivery of missing ingredients and equipment

[1739] The server checks whether the user has all the ingredients and equipment required for the generated recipe. If there are any shortages, it automatically generates a delivery request and sends an order to an external supply service. The input is the recipe data and the current list of ingredients and equipment, and the shortages are clearly displayed as output. Based on this shortage information, an order is sent to the delivery service, and the necessary ingredients and equipment are delivered to the user.

[1740] Step 7: Displaying results and cooking assistance

[1741] The server sends the final recipe data, cooking steps, and information on missing ingredients and equipment to the device. The device then displays this information in an easy-to-understand manner, providing text and visual content to guide the user. For example, it displays detailed steps, cooking time, and a list of necessary equipment for making "Oyakodon." It also provides relaxing music and video links depending on the user's emotional state. The input data is the recipe details and guide information, and by displaying and playing this information, the device allows the user to smoothly proceed with cooking.

[1742] (Application example 2)

[1743] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1744] Conventional recipe suggestion systems do not fully consider the use of food in the refrigerator or the user's emotional state, making it difficult to provide a wide variety of recipe suggestions or services that respond to the user's emotions.In addition, in physical stores, there is a lack of services that suggest real-time recipes based on the food items that the user plans to purchase, or that encourage the purchase of necessary food and equipment, making it difficult to improve user satisfaction.

[1745] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating recipes based on the acquired food list and preference information, means for identifying and recognizing foods in the refrigerator through image analysis, means for referencing past usage data and personalizing recipes based on the user's preferences, means for automatically generating and sending a delivery request when necessary foods and utensils are missing, means for recognizing the user's emotional state and adjusting the recipe based on the emotional state, means for photographing foods to be purchased in the store and suggesting recipes, means for suggesting purchases when necessary foods and utensils are missing, and means for providing the user with recipes and cooking instructions. This enables a wide variety of recipes to be suggested based on the user's emotional state and the foods to be purchased, providing a comfortable shopping and cooking experience in a physical store.

[1746] "Acquired food list and preference information" refers to information indicating the food items currently owned by the user and the user's food preferences.

[1747] "Image analysis" is the technique of analyzing digital images to identify specific objects and extract information.

[1748] "Food in refrigerator" means food currently in a user's refrigerated storage unit.

[1749] "Past Usage Data" means historical information and related data generated by a User's use of the System.

[1750] "User Preferences" is information that indicates a user's personal preferences regarding foods and recipes that they like.

[1751] "Personalizing recipes" means adjusting recipes based on the user's individual preferences and past usage data, and suggesting recipes in the most optimal form for each individual user.

[1752] A "delivery request" is order information for delivery of necessary food and equipment to a specified location.

[1753] "User's emotional state" refers to the user's mental and emotional state as determined by analyzing voice and facial expression data.

[1754] "Foods to be purchased" refers to foods that a user intends to purchase in a physical store.

[1755] "Real-time recipe suggestions" means instantly generating and suggesting recipes based on the current situation and user input.

[1756] "Cooking instructions" are the detailed instructions and steps required to prepare a particular food product.

[1757] The system for realizing this invention is mainly composed of a server, a terminal, and a user. Each component and its operation will be explained below.

[1758] 1. Server Functions

[1759] The server has the following functions:

[1760] A means for generating recipes based on the obtained food list and preference information:

[1761] The server generates appropriate recipes based on the food list and cooking preferences entered or photographed by the user, and uses a generative AI model to suggest recipes that best match the ingredients and preferences.

[1762] Image analysis to identify and identify food items in refrigerators:

[1763] The system receives image data of the refrigerator sent by the user and uses an image analysis algorithm (e.g., Google Cloud Vision API) to identify the type and amount of food.

[1764] Ways to personalize recipes based on user preferences using past usage data:

[1765] The server uses past usage history data to suggest recipes tailored to the user's individual preferences, prioritizing recipes that have been popular in the past and ingredients that are frequently used.

[1766] A way to automatically generate and send delivery requests when needed food and equipment are in short supply:

[1767] If the server is running low on ingredients or cooking equipment required for the generated recipe, it will automatically send an order in conjunction with a delivery service and arrange for prompt delivery.

[1768] A way to recognize the user's emotional state and adjust recipes based on that emotional state:

[1769] The server analyzes voice and facial expression data to recognize the user's emotional state. For example, if the user is feeling stressed, it will prioritize recipes that are easy to prepare and have a relaxing effect.

[1770] A way to take a photo of the food you plan to buy in the store and get recipe suggestions:

[1771] The server takes a photo of the food the user plans to buy in a physical store, analyzes the image, and suggests suitable recipes, supporting the user's shopping and increasing their desire to buy.

[1772] Providing suggested purchases of necessary food and equipment if they are in short supply:

[1773] If the user is running low on food or equipment that he or she plans to purchase, the server will indicate this and suggest the purchase.

[1774] 2. Device Features

[1775] The device (e.g., smartphone, tablet) has the following functions:

[1776] Getting user input:

[1777] It offers a variety of input methods, including taking pictures of the refrigerator, entering a food list into a text box, and collecting voice and facial expression data.

[1778] Result display and cooking assistance:

[1779] The system supports cooking by displaying recipe data and cooking instructions sent from the server in an easy-to-understand manner to the user, and also provides relaxing music and video links as needed.

[1780] 3. User Behavior

[1781] Users use the system to perform the following actions:

[1782] Take and upload a picture of the food in your refrigerator.

[1783] Enter your ingredient list and cooking preferences into the text boxes.

[1784] Provides voice and facial expression data.

[1785] Order the necessary ingredients and equipment based on the proposed recipe.

[1786] Examples of specific examples and prompts

[1787] For example, if a user takes a photo of "tomatoes, cheese, pasta" in the refrigerator and the foods they have purchased, and provides the audio data to the app:

[1788] The server will parse the message assuming the following prompt:

[1789] Example prompt sentence:

[1790] "Generate the best recipe from the ingredients you have based on the user's emotional state. The current ingredients are tomatoes, cheese, and pasta."

[1791] This allows for flexible and personalized recipe suggestions tailored to the user's needs, providing a seamless shopping and cooking experience even in physical stores.

[1792] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1793] Step 1:

[1794] The user uses a smartphone or tablet to take pictures of the food in the refrigerator or the food they plan to buy. The user also inputs the food list into a text box and provides voice and facial expression data. The input data consists of image data, text data, and voice and facial expression data.

[1795] Step 2:

[1796] The device sends the collected image data, text data, and voice and facial expression data to the server, where it is entered as user input data.

[1797] Step 3:

[1798] The server uses image analysis algorithms to identify and identify the food in the refrigerator or the food you plan to buy from the image data sent to it. The extracted food data is output. At this stage, specific food names such as "tomato," "cheese," and "pasta" are obtained from the image data.

[1799] Step 4:

[1800] The server analyzes the voice and facial expression data to recognize the user's emotional state. It uses an emotion recognition engine (e.g., Google Cloud Speech-to-Text) to perform the analysis and outputs the user's emotional state. For example, it determines whether the user is relaxed or stressed.

[1801] Step 5:

[1802] The server uses a generative AI model based on the acquired food list and preference information to generate recipes. The input is food data and emotional state data, and the output is a suggested recipe. At this stage, a specific recipe such as "pasta with tomato and cheese" is output.

[1803] Step 6:

[1804] The server references past usage data and personalizes recipes based on the user's preferences. Entering past usage data results in personalized recipes. For example, if a user has previously preferred "tomato pasta," the recipe will be adjusted to take this into account.

[1805] Step 7:

[1806] The server automatically generates and sends a delivery request if the food and equipment required for the generated recipe are missing. The input is the missing food and equipment data, and the output is the generated delivery request. For example, if "cheese" is missing, a request is generated suggesting the purchase of that item.

[1807] Step 8:

[1808] The server sends the final recipe data, cooking instructions, and information on missing foods and equipment to the device. The input is the generated recipe and cooking instructions data, and the output is the information sent to the device. Specifically, the step-by-step cooking instructions and instructions for purchasing "cheese" are displayed.

[1809] Step 9:

[1810] The device finally displays the recipe and cooking instructions to the user, and provides relaxing music and video links as needed.The user can then cook based on the displayed information and enjoy the suggested recipe.

[1811] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1812] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1813] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1814] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1815] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1816] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1817] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1818] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1819] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1820] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1821] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1822] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the prese...

Claims

1. means for generating recipes based on the obtained food list and preference information; means for identifying and recognizing food items in the refrigerator through image analysis; A way to reference past usage data and personalize recipes based on user preferences; means for automatically generating and sending delivery requests when necessary food and equipment are in short supply; a means for providing recipes and cooking instructions to users; A system including:

2. 2. The system according to claim 1, further comprising means for making suggestions to minimize food waste, taking into account expiration dates of food and appliances.

3. 10. The system of claim 1, further comprising means for receiving and analyzing user input as textual and image data.

Images