System

Abstract

To provide a system for enabling a user to obtain a well-balanced menu by a simple step, to reduce the waste of food materials, and to plan healthy and various meals which can be enjoyed by all the family members.SOLUTION: A system comprising: image recognition means for identifying food in a refrigerator of a user; means for integrating the identified food information with a user profile; means for generating a menu and recipe based on the food information and the user profile; means for generating information on serving methods and cutting methods related to the menu and the recipe; means for transmitting the generated information to a user terminal; means for listing up missing food and automatically generating a shopping list; means for analyzing tastes and nutritional needs of all family members and proposing a balanced diet; and means for proposing a drink pairing and a dining menu.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's world, it can be difficult to plan and come up with ideas for meals, and busy lifestyles mean limited time for planning nutritionally balanced meals. In addition, people tend to neglect managing and utilizing ingredients in their refrigerators, resulting in food waste. Another challenge is the difficulty of providing balanced meals that meet the preferences and nutritional needs of each member of the family. The present invention aims to solve these problems, reduce food waste, and provide healthy and diverse meals. [Means for solving the problem]

[0005] The present invention provides a system including an image recognition means for identifying ingredients in a user's refrigerator, a means for integrating the identified ingredient information with a user profile, a means for generating menus and recipes based on the ingredient information and the user profile, a means for generating information on plating and cutting methods related to the menus and recipes, a means for sending the generated information to a user terminal, a means for listing missing ingredients and automatically generating a shopping list, a means for analyzing the preferences and nutritional needs of all family members and suggesting balanced meals, and a means for suggesting drink pairings and restaurant menus.

[0006] This allows users to create balanced meals in simple steps, reduce food waste, and plan healthy and varied meals the whole family can enjoy.

[0007] "Image recognition means" refers to technology or algorithms for analyzing image data and identifying specific objects or information.

[0008] A "user profile" is a data set that includes a user's personal information, preferences, health status, past dietary history, etc.

[0009] A "menu" refers to a menu combination for a particular meal or dish.

[0010] A "recipe" is a list of steps and ingredients needed to make a particular dish.

[0011] "Plating" refers to the techniques and ideas for arranging food beautifully on a plate.

[0012] "Cutting method" refers to the technique or method of cutting ingredients.

[0013] "User terminal" refers to an electronic device used by a user, such as a smartphone, tablet, or PC.

[0014] A "shopping list" refers to a list of ingredients or items that need to be purchased.

[0015] "Preferences" refers to the user's favorite ingredients and cooking tendencies.

[0016] "Nutritional needs" refers to the nutrient requirements needed for a particular health condition or purpose. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention is a system for managing ingredients stored in a user's refrigerator and proposing menus and recipes based on the ingredients, and is comprised of a server and a user terminal. Specific embodiments of the present invention will be described below.

[0039] System Configuration

[0040] The server is a back-end system that includes image recognition, a database, recipe generation, plating assistance, shopping list generation, and preference analysis. The server processes data sent from the user's device and generates the necessary information.

[0041] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server, and corresponds to a smartphone, tablet, etc. An application is installed on the terminal, and the user interacts with the system through this application.

[0042] Program processing overview

[0043] First, the user takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. At this time, it uses an image recognition algorithm to extract the name and quantity of each ingredient and stores them in a database.

[0044] Next, the user inputs their preferences, health status, and dietary information via their device. This information is sent to the server, which updates the user profile. The server then generates optimal menus and recipes based on the user profile and the identified ingredients.

[0045] The generated menus and recipes include both those that use only available ingredients and those that can be realized by supplementing ingredients that are in short supply. In addition, suggestions are also made on how to present the food after cooking and how to cut the ingredients, and this information is sent to the user's device.

[0046] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[0047] Finally, the server analyzes the preferences and nutritional needs of each family member and suggests balanced meals that everyone can enjoy. This information is also sent to the device, allowing the user to easily implement the suggested menu. The server also provides information on drink pairings, dining out menus, and restaurant recommendations.

[0048] Specific examples

[0049] For example, a user takes a photo of the inside of their refrigerator with their smartphone and uploads it to a server through an application. The server identifies the presence of tomatoes, chicken, and spinach in the image. The user then enters their personal information, such as "I want a low-carb diet" and "I like tomato dishes."

[0050] The server uses this information to suggest low-carb recipes, such as tomato chicken stew, and provides presentation and cutting instructions. If you need more cream than the spinach you need, the server adds it to your shopping list and notifies you.

[0051] Users can follow suggested recipes and beautifully arrange the food on a plate using the provided plating methods. They can also get recommendations for white wine pairings and information on nearby healthy restaurants.

[0052] In this way, the system of the present invention supports the user in planning their meals and provides a healthy and varied diet.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user takes a photo of the food items in the refrigerator using their smartphone (terminal).

[0056] Specific operation: The user launches the camera app on their smartphone and takes a picture of the ingredients in the refrigerator.

[0057] Step 2:

[0058] The terminal uploads the captured photos to the server.

[0059] Specific operation: The device sends the captured photo data to the specified API endpoint.

[0060] Step 3:

[0061] The server analyzes the received image data and identifies the ingredients.

[0062] Specific operation: The server uses an image recognition algorithm (e.g., Convolutional Neural Network) to identify the type and quantity of ingredients.

[0063] Step 4:

[0064] The server stores the identified ingredient information in a database.

[0065] Specific operation: The extracted list of ingredients and their quantities are associated with the user's profile and recorded in the database.

[0066] Step 5:

[0067] Users input their preferences, health status, and food intake information via the terminal.

[0068] Specific operation: The user enters various information into a form within the application and presses the submit button.

[0069] Step 6:

[0070] The terminal transmits the input user information to the server.

[0071] Specific operation: The device sends the entered preference and health status data to the server via API.

[0072] Step 7:

[0073] The server adds or updates the received user information to the profile.

[0074] Specific operations: The server merges the existing user profile with the newly received information and updates the database.

[0075] Step 8:

[0076] The server integrates the information about ingredients in the refrigerator with the updated user profile to generate menus and recipes.

[0077] What it does: Uses an AI recommendation system to suggest optimal menus and recipes based on user profile and available ingredients.

[0078] Step 9:

[0079] The server sends the generated menu and recipes to the terminal.

[0080] Specific operation: The proposed recipe and its cooking instructions are sent to the device in JSON format via API.

[0081] Step 10:

[0082] The server generates images of how to arrange and cut the food based on the proposed menu.

[0083] Specific operations: Generate presentation guides and visual materials after cooking and prepare them as image data.

[0084] Step 11:

[0085] The server sends the presentation image and guide information to the terminal.

[0086] Specific operation: The generated presentation guide and images are sent to the device via API.

[0087] Step 12:

[0088] The server lists the ingredients that are missing and generates a shopping list.

[0089] Specific behavior: Compare the inventory in the refrigerator with the suggested recipe and make a list of missing ingredients.

[0090] Step 13:

[0091] The server transmits the shopping list to the terminal.

[0092] Specific operation: The list of ingredients that are in short supply and their purchase links are sent to the device via API.

[0093] Step 14:

[0094] The server analyzes the preferences and nutritional needs of each family member and generates a balanced menu.

[0095] Specific operation: The AI ​​references each user profile and generates a menu that can be enjoyed by all.

[0096] Step 15:

[0097] The server sends the proposed balanced menu to the terminal.

[0098] Specific operation: The generated menu and its procedures are sent to the device via API.

[0099] Step 16:

[0100] The server will suggest drink pairings to go with the menu, dining out menus, and recommended restaurants.

[0101] What it does: Recommends drinks and dining options based on a suggested menu and creates a list.

[0102] Step 17:

[0103] The server transmits this additional information to the terminal and notifies the user.

[0104] Specific operation: Recommendation information is sent to the device via API and notified to the user.

[0105] Example 1

[0106] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0107] In today's busy lifestyles, efficiently managing ingredients and planning meals is a difficult task. Properly managing ingredients in the refrigerator and providing menus and recipes that match the user's preferences and health status are important for achieving a healthy and diverse diet. However, conventional systems do not adequately address these needs, forcing users to spend time managing ingredients and searching for recipes. Furthermore, it is difficult to provide suggestions that balance the preferences and nutritional needs of each family member.

[0108] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0109] In this invention, the server includes an image recognition unit for identifying ingredients in the user's refrigerator, a unit for integrating the identified ingredient information with a user profile, a unit using a generative AI model to generate menus and recipes based on the ingredient information and the user profile, a unit using an image generation algorithm to generate information on presentation and cutting methods related to the menus and recipes, a unit for sending the generated information to a user terminal, a unit for listing missing ingredients and automatically generating a shopping list, a unit for analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, and a unit for suggesting drink pairings, restaurant menus, and recommended restaurants. This allows for efficient management of ingredients in the refrigerator and automatically generating optimal menus and recipes tailored to the user's preferences and health status. It also provides balanced meal suggestions and automatic purchasing assistance for missing ingredients.

[0110] "Image recognition means" refers to means that includes algorithms and techniques for identifying ingredients in the refrigerator.

[0111] A "user profile" is a collection of personal information related to a user, such as the user's health status, preferences, and seasonal information.

[0112] A "generative AI model" is an artificial intelligence model that automatically generates menus and recipes based on ingredient information and user profiles.

[0113] An "image generation algorithm" is an algorithm for generating information on presentation and cutting methods related to menus and recipes in the form of realistic images.

[0114] A "user terminal" is a device that a user uses to interact with the system, such as a smartphone or tablet.

[0115] The "shopping list generating means" is a means for listing ingredients that are in short supply and automatically generating a shopping list.

[0116] The "preference analysis tool" is a tool for analyzing the preferences and nutritional needs of each family member and proposing balanced meals.

[0117] The "pairing suggestion means" is a means for suggesting drink pairings, dining out menus, and information on recommended restaurants.

[0118] MODE FOR CARRYING OUT THE INVENTION

[0119] The present invention is a system that manages ingredients stored in a user's refrigerator and suggests menus and recipes based on the ingredients. The system is composed of a server and a user terminal. Specific embodiments of the present invention are described below.

[0120] The server is a backend system that includes image recognition means, a database, a recipe generation means, a plating assistance means, a shopping list generation means, and a preference analysis means. The server processes data sent from the user's device and generates the necessary information. The server hardware is a general-purpose server machine (e.g., a server equipped with a high-performance CPU, sufficient memory, and storage). The software used includes image recognition algorithms such as "YOLO" or "Google (registered trademark) Cloud Vision API," databases such as "MySQL (registered trademark)," "Firebase," or "AWS (registered trademark) DynamoDB," and generative AI models such as "GPT-3 (registered trademark)."

[0121] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server; it can be a smartphone or tablet. A dedicated application is installed on the terminal, and the user interacts with the system through this application. This application has functions for taking pictures, entering data, and receiving notifications. The terminal hardware is a smartphone with a camera function (e.g., ANDROID (registered trademark) or iPhone (registered trademark)). The software is a dedicated mobile application. This app provides a user interface that allows the user to upload images, enter profile information, and receive notifications.

[0122] The user first takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. To do this, the user opens the refrigerator door and takes a photo of the entire interior with the smartphone camera. Next, the user taps the "Upload" button on the application to send the image. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. For the analysis, it uses the image recognition algorithm "YOLO" and "Google Cloud Vision API." The server then stores the names and quantities of the identified ingredients in a database. This database uses "MySQL," "Firebase," or "AWS DynamoDB."

[0123] Next, the user enters their preferences, health status, and dietary information via their device. For example, if the user prefers a "low-carb" or "low-fat" diet, they enter this information into a form within the app and tap the "Submit" button. The server receives this information and updates the user profile. The server then generates optimal menus and recipes based on the updated user profile and the identified ingredient information. In this process, a generative AI model such as "GPT-3" is used to generate recipes that take into account the user's preferences and available ingredients.

[0124] In addition to the recipe, the server also suggests how to arrange the food after cooking and how to cut the ingredients. To do this, it uses an "image generation AI model" (e.g., DALL-E) to generate a sample image of the presentation and sends it to the user's device. If any ingredients are missing, the server lists them and automatically generates a shopping list. This shopping list also includes a link to purchase the food online and notifies the user.

[0125] The server also analyzes the preferences and nutritional needs of each family member and suggests balanced meals. This analysis could potentially use the Human Nutrition API. It also provides information on drink pairings, dining out menus, and recommended restaurants. Drink pairing information can be obtained from the Wine.com API, while restaurant information can be obtained from the Google Places API or Yelp API.

[0126] Specific examples

[0127] For example, a user can take a photo of the contents of their refrigerator with their smartphone and upload it to a server through an application. The server can identify the presence of tomatoes, chicken, and spinach in the image. The user then enters information into the app, such as "I'm looking for a low-carb diet" and "I like tomato dishes." Based on this information, the server can suggest a low-carb recipe for tomato chicken stew. If the user has the spinach needed for this recipe in their refrigerator but is short on cream, the server will add the cream to their shopping list and notify them. The user can then cook the dish according to the suggested recipe and arrange it on a plate using the suggested plating method. The server can also suggest white wine pairings and provide information about nearby healthy restaurants.

[0128] Prompt example

[0129] "You take a photo of the contents of your refrigerator and upload it to the system. It identifies that you have tomatoes, chicken, and spinach, and requests a low-carb meal and a tomato dish."

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Program processing flow

[0132] Step 1:

[0133] Users take photos of the inside of the refrigerator with their smartphone (device) and upload them to the server through the application. Specifically, the user opens the refrigerator door and takes a photo of the entire interior with the smartphone camera. Next, they tap the "upload" button in the application to send the image.

[0134] Input: Image of the inside of the refrigerator

[0135] Output: Image data sent to the server

[0136] Step 2:

[0137] The server analyzes the received images and identifies the ingredients in the refrigerator in real time. It uses an image recognition algorithm (e.g., YOLO or Google Cloud Vision API) to extract the names and quantities of ingredients and store them in a database. Specifically, after receiving the images, the server runs the image recognition software and stores the identified ingredient data in MySQL, Firebase, or AWS DynamoDB.

[0138] Input: Image data uploaded by the user

[0139] Output: Database update information including ingredient names and quantities

[0140] Step 3:

[0141] Users input their preferences, health status, and dietary information via their device. Specifically, the user enters information such as "low-carb" or "low-fat" into the application's input form and taps the "Submit" button. The server receives this information and updates the user profile.

[0142] Input: User preferences and health data

[0143] Output: Updated user profile

[0144] Step 4:

[0145] The server generates optimal menus and recipes based on the updated user profile and identified ingredient information. Specifically, the server uses a generative AI model (e.g., GPT-3) to generate recipes that take into account the user's preferences and available ingredients.

[0146] Input: Updated user profile and ingredient information

[0147] Output: Generated menu and recipes

[0148] Step 5:

[0149] The server then suggests post-cooking plating and cutting methods related to the generated menu and recipe. Specifically, the server uses an image generation AI model (e.g., DALL-E) to generate sample plating images and sends this information to the user's device.

[0150] Input: Generated menu and recipe

[0151] Output: Suggestions for presentation and cutting of ingredients

[0152] Step 6:

[0153] The server lists the ingredients that are in short supply and automatically generates a shopping list. Specifically, the server checks for ingredients that are in short supply, creates a shopping list using the Amazon Web Services Lambda function, and notifies the user with a link to purchase the ingredients.

[0154] Input: Ingredient information from the database and the generated recipe

[0155] Output: List of ingredients that are in short supply and a shopping list

[0156] Step 7:

[0157] The server analyzes the preferences and nutritional needs of each family member and suggests balanced meals. Specifically, the server uses the "Human Nutrition API" to calculate nutritional information and compare it with the preference information of each family member. In addition, the server also provides drink pairings and recommended restaurant information.

[0158] Input: Data on the preferences and health status of all family members

[0159] Output: Balanced meal suggestions, drink pairings and restaurant recommendations

[0160] (Application example 1)

[0161] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0162] Conventional food management systems were able to identify ingredients in the refrigerator and suggest recipes based on the user's preferences and health information. However, they were unable to automatically process the entire process, from purchasing missing ingredients to delivery. Furthermore, they lacked optimal recipe suggestions using generative AI models, making it impossible to achieve the high level of personalization desired by users. The present invention aims to solve these issues and provide users with a consistent, personalized meal suggestion and delivery service.

[0163] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0164] In this invention, the server includes an image recognition unit for identifying ingredients in the user's refrigerator, a unit for integrating the identified ingredient information with a user profile, a unit for generating menus and recipes based on the ingredient information and the user profile, a unit for generating information on plating and cutting methods related to the menus and recipes, a unit for sending the generated information to a user terminal, a unit for listing missing ingredients and automatically generating a shopping list, a unit for analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, a unit for suggesting drink pairings and restaurant menus, a unit for automatically selecting missing ingredients based on the identified ingredient information and delivering them together with cooking, and a unit for generating optimal dishes and recipes using a generative AI model. This allows users to manage ingredients in their refrigerator in real time, receive automatic delivery of missing ingredients, and enjoy highly personalized menus and dishes.

[0165] "Image recognition means for identifying ingredients in a user's refrigerator" refers to technology for analyzing images of the inside of a refrigerator taken by a user and identifying ingredients.

[0166] "Means for integrating the identified ingredient information with a user profile" refers to a method for integrating ingredient information identified in a refrigerator with user preferences, health information, etc.

[0167] "Means for generating menus and recipes based on the ingredient information and user profile" refers to technology for generating optimal menus and recipes based on identified ingredient information and user profile information.

[0168] The term "means for generating information on presentation and cutting methods related to the menu and recipe" refers to a method for generating information on presentation and cutting methods for the proposed menu and recipe.

[0169] "Means for transmitting the generated information to the user terminal" refers to a communication method for transmitting information such as the generated menu, recipes, and presentation methods to the user terminal.

[0170] "Means for listing missing ingredients and automatically generating a shopping list" refers to a method for listing missing ingredients required for a proposed recipe and automatically generating a shopping list.

[0171] "Means for analyzing the preferences and nutritional needs of all family members and suggesting balanced meals" refers to a method for analyzing the preferences and nutritional needs of all family members and suggesting balanced meals based on that analysis.

[0172] "Means for suggesting drink pairings and restaurant menus" refers to methods for recommending drinks that go well with the proposed dishes and restaurant menus.

[0173] "Means for automatically picking up missing ingredients based on the identified ingredient information and delivering them together with cooking" refers to a method for automatically picking up missing ingredients based on identified ingredient information and delivering them together with cooking.

[0174] "Means for generating optimal dishes and recipes using a generative AI model" refers to technology that uses a generative AI model to generate optimal dishes and recipes based on the user's preferences and information about ingredients in the refrigerator.

[0175] The present invention relates to a system that manages ingredients in a user's refrigerator and proposes menus and recipes based on the user's preferences and health information. This allows the user to receive personalized meal suggestions, including automatic delivery of ingredients that the user is running low on. Specific embodiments of the present invention are described below.

[0176] System Configuration

[0177] This system mainly consists of a server and a terminal.

[0178] 1. Server:

[0179] The server is a backend system that includes an image recognition means, a database, a recipe generation means, a plating support means, a shopping list generation means, a preference analysis means, and a generative AI model.

[0180] The server has the function of processing data sent from the user's terminal and generating the necessary information.

[0181] 2. Terminal:

[0182] The terminal is a device that allows the user to take a photo of the inside of the refrigerator and send it to the server. This can be a smartphone or tablet.

[0183] Applications are installed on the terminals, and users interact with the system through these applications.

[0184] Program processing overview

[0185] 1. Image Recognition Methods:

[0186] First, the user takes a photo of the inside of the refrigerator using their smartphone (terminal) and uploads it to the system.

[0187] The server analyzes the received images and identifies the ingredients in the refrigerator in real time, using an image recognition algorithm (e.g., TENSORFLOW®) to extract the name and quantity of each ingredient and store them in a database.

[0188] 2. Data integration and recipe generation:

[0189] Users input their preferences, health status, and dietary information through their device, which is then sent to the server and the user profile is updated.

[0190] The server generates optimal menus and recipes based on the user profile and identified ingredient information. It generates dishes and recipes using a generative AI model (e.g., a generative AI model).

[0191] 3. Presentation and Delivery:

[0192] The generated menus and recipes include both those that use only available ingredients and those that can be achieved by supplementing missing ingredients.

[0193] Suggestions are also given on how to arrange the food after cooking and how to cut the ingredients, and this information is sent to the user's terminal.

[0194] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[0195] The server also provides a means to automatically pick up missing ingredients and deliver them along with cooking.

[0196] 4. Meeting the tastes and nutritional needs of the whole family:

[0197] It analyzes the preferences and nutritional needs of each family member and suggests balanced meals that everyone can enjoy. This information is also sent to the device, allowing the user to easily put the suggested menu into practice.

[0198] The server also provides information on drink pairings, dining out menus, and recommended restaurants.

[0199] Specific examples

[0200] For example, a user can take a photo of the contents of their refrigerator with their smartphone and upload it to a server through an application. The server can identify the presence of tomatoes, chicken, and spinach in the image. The user then inputs their personal information, such as "I'm looking for a low-carb diet" and "I like tomato dishes." The server then uses this information to suggest low-carb recipes, such as tomato chicken stew. It also provides an image of how the food will look after cooking and how to cut it. If the user needs more cream than the spinach they need, the server adds it to their shopping list and notifies them. The user can then prepare the dish according to the suggested recipe and beautifully arrange it on a plate using the provided plating method. They can also receive recommendations for white wines to pair with their drinks and information about nearby healthy restaurants.

[0201] Prompt Sentence Examples

[0202] The following prompt sentence is used as input to the generative AI model:

[0203] I have tomatoes and chicken in my fridge. Analyze images of the fridge and suggest low-calorie meals using these ingredients. If the suggested meal contains missing ingredients, add them to the shopping list and offer them as a delivery option.

[0204] The above is an embodiment of the present invention, which allows users to efficiently manage ingredients in their refrigerators and receive personalized meal suggestions and delivery services.

[0205] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0206] Step 1:

[0207] The user takes a photo of the inside of the refrigerator using their smartphone (device). The image is saved on the device and becomes the input data for the system.

[0208] Step 2:

[0209] The user uploads a photo of the inside of the refrigerator to the server via the terminal. The terminal sends the image data to the server via the Internet. In this case, the image file becomes the input data.

[0210] Step 3:

[0211] The server analyzes the received images and identifies the ingredients in the refrigerator in real time. Specifically, the server uses an image recognition algorithm (e.g., TensorFlow) to extract the name and quantity of each ingredient and store the identified ingredient information in a database. In this process, image data is the input and identified ingredient information is the output.

[0212] Step 4:

[0213] Users input their preferences, health status, and dietary information through their device. This information is sent from the device to the server, where the user profile is updated. Here, preferences and health information are input data, and the updated user profile is output.

[0214] Step 5:

[0215] The server generates optimal menus and recipes based on the user profile and identified ingredient information. The server uses a generative AI model (e.g., generative AI model) to generate dishes and recipes based on the prompt. In this process, the user profile and ingredient information are input data, and the generated menus and recipes are output.

[0216] Step 6:

[0217] The server generates information on how to arrange and cut ingredients related to the menu and recipe, allowing the user to receive specific cooking and presentation advice. In this process, the menu and recipe are input data, and information on how to arrange and cut ingredients is output.

[0218] Step 7:

[0219] The server sends the generated information to the user's terminal, where the user can view the proposed menu, recipes, presentation methods, and other information. In this process, all the information generated up to that point is input data, and the data sent to the user's terminal is the output.

[0220] Step 8:

[0221] The server lists the missing ingredients and automatically generates a shopping list. Furthermore, this list also includes a purchase link so that the user can easily purchase the missing ingredients. In this process, the required ingredient information from the menu and recipes is the input data, and the shopping list is the output.

[0222] Step 9:

[0223] The server automatically picks up missing ingredients based on the identified ingredient information, cooks them, and delivers them. Specifically, an order is issued to a partner delivery service. In this process, the missing ingredient information is the input data, and a delivery order is the output.

[0224] Step 10:

[0225] The server analyzes the preferences and nutritional needs of each family member and suggests balanced meals. It also suggests drink pairings and restaurant menus, and users can receive this information on their devices. In this process, the input data is the profiles of each family member, and the output is balanced meals and pairing information.

[0226] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0227] This system manages the ingredients in a user's refrigerator and suggests menus and recipes based on that information. In particular, by combining it with an emotion engine, this system makes it possible to suggest meals based on the user's emotions. It is composed of a server and a user terminal.

[0228] System Configuration

[0229] The server is a back-end system that includes image recognition, a database, recipe generation, plating assistance, shopping list generation, preference analysis, and an emotion engine. The server processes data sent from the user's device and generates the necessary information.

[0230] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server, and corresponds to a smartphone, tablet, etc. An application is installed on the terminal, and the user interacts with the system through this application.

[0231] Program processing overview

[0232] First, the user takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. At this time, it uses an image recognition algorithm to extract the name and quantity of each ingredient and stores them in a database.

[0233] Next, the user inputs their preferences, health status, and dietary information via their device. This information is sent to the server, which updates the user profile. The server then generates optimal menus and recipes based on the user profile and the identified ingredients.

[0234] The generated menus and recipes include both those that use only available ingredients and those that can be realized by supplementing ingredients that are in short supply. In addition, suggestions are also made on how to present the food after cooking and how to cut the ingredients, and this information is sent to the user's device.

[0235] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[0236] The server analyzes the preferences and nutritional needs of each family member to suggest balanced meals that everyone can enjoy. This information is also sent to the device, allowing users to easily implement the suggested menu. The server also provides information on drink pairings, dining out menus, and recommended restaurants.

[0237] Emotion engine integration

[0238] This system uses an emotion engine to recognize the user's emotions in real time. The emotion engine can detect the user's emotional state through voice input and facial expression recognition. This emotional information is directly reflected in the menu and recipe suggestions generated by the server.

[0239] For example, if a user is feeling down, the emotion engine can detect this information and suggest colorful, nutritious dishes to lift their spirits. On the other hand, if the user is tired, it can suggest recipes that are easy to prepare or that use ingredients that have a relaxing effect. In this way, personalized meal suggestions based on the emotion engine are possible.

[0240] Specific examples

[0241] For example, a user can take a photo of the inside of their refrigerator with their smartphone and upload it to a server through an application, which can then identify the presence of tomatoes, chicken, and spinach in the image.

[0242] Next, the user inputs their preferences, such as "I want a low-carb diet" and "I like tomato dishes." The emotion engine also detects emotional information from the user's voice input, such as "I'm feeling a little stressed."

[0243] The server uses this information to suggest low-carb recipes, such as tomato chicken stew, that contain ingredients that have stress-reducing properties. The suggestions also include presentation and cutting instructions. If cream is the missing ingredient, the server adds it to the shopping list and notifies the user.

[0244] Users can cook according to the suggested recipes and beautifully arrange the food on a plate using the provided plating methods. They can also get information on relaxing herbal teas to pair with their drinks, as well as healthy restaurants nearby.

[0245] In this way, the system of the present invention integrates the user's emotions with the management of ingredients in the refrigerator, enabling the provision of personalized, healthy, and diverse meals.

[0246] The processing flow will be explained below.

[0247] Step 1:

[0248] The user takes a photo of the food items in the refrigerator using their smartphone (terminal).

[0249] Specific operation: The user launches the camera app on their smartphone and takes a picture of the ingredients in the refrigerator.

[0250] Step 2:

[0251] The terminal uploads the captured photos to the server.

[0252] Specific operation: The device sends the captured photo data to the specified API endpoint.

[0253] Step 3:

[0254] The server analyzes the received image data and identifies the ingredients.

[0255] Specific operation: The server uses an image recognition algorithm (e.g., Convolutional Neural Network) to identify the type and quantity of ingredients.

[0256] Step 4:

[0257] The server stores the identified ingredient information in a database.

[0258] Specific operation: The extracted list of ingredients and their quantities are associated with the user's profile and recorded in the database.

[0259] Step 5:

[0260] Users input their preferences, health status, and food intake information via the terminal.

[0261] Specific operation: The user enters various information into a form within the application and presses the submit button.

[0262] Step 6:

[0263] The terminal transmits the input user information to the server.

[0264] Specific operation: The device sends the entered preference and health status data to the server via API.

[0265] Step 7:

[0266] The server adds or updates the received user information to the profile.

[0267] Specific operations: The server merges the existing user profile with the newly received information and updates the database.

[0268] Step 8:

[0269] The device collects voice input and facial expression images to detect the user's emotions.

[0270] Specific operation: The terminal records the user's voice or saves a photograph of the user's face.

[0271] Step 9:

[0272] The terminal uses an emotion engine to analyze the user's emotions.

[0273] Specific behavior: Estimates the user's emotional state using voice and facial expression analysis algorithms.

[0274] Step 10:

[0275] The terminal transmits the analysis results to the server.

[0276] Specific operation: The estimated emotion information is sent to the server via API.

[0277] Step 11:

[0278] The server integrates information about ingredients in the refrigerator with the updated user profile and emotional information to generate menus and recipes.

[0279] What it does: Uses an AI recommendation system to suggest optimal menus and recipes based on user profile, available ingredients, and emotional information.

[0280] Step 12:

[0281] The server sends the generated menu and recipes to the terminal.

[0282] Specific operation: The proposed recipe and its cooking instructions are sent to the device in JSON format via API.

[0283] Step 13:

[0284] The server generates images of how to arrange and cut the food based on the proposed menu.

[0285] Specific operations: Generate presentation guides and visual materials after cooking and prepare them as image data.

[0286] Step 14:

[0287] The server sends the presentation image and guide information to the terminal.

[0288] Specific operation: The generated presentation guide and images are sent to the device via API.

[0289] Step 15:

[0290] The server lists the ingredients that are missing and generates a shopping list.

[0291] Specific behavior: Compare the inventory in the refrigerator with the suggested recipe and make a list of missing ingredients.

[0292] Step 16:

[0293] The server transmits the shopping list to the terminal.

[0294] Specific operation: The list of ingredients that are in short supply and their purchase links are sent to the device via API.

[0295] Step 17:

[0296] The server analyzes the preferences and nutritional needs of each family member and generates a balanced menu.

[0297] Specific operation: The AI ​​references each user profile and generates a menu that can be enjoyed by all.

[0298] Step 18:

[0299] The server sends the proposed balanced menu to the terminal.

[0300] Specific operation: The generated menu and its procedures are sent to the device via API.

[0301] Step 19:

[0302] The server will suggest drink pairings to go with the menu, dining out menus, and recommended restaurants.

[0303] What it does: Recommends drinks and dining options based on a suggested menu and creates a list.

[0304] Step 20:

[0305] The server transmits this additional information to the terminal and notifies the user.

[0306] Specific operation: Recommendation information is sent to the device via API and notified to the user.

[0307] Example 2

[0308] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0309] In modern life, there is a demand for efficient management of ingredients in the refrigerator and for optimal menus and recipes to be proposed based on the user's preferences and health status. However, conventional systems have limitations in ingredient management and menu proposals, and are unable to realize personalized meal proposals that reflect the user's emotional state. Furthermore, there are insufficient means for grasping ingredient inventory status and efficiently replenishing ingredients that are in short supply. This leads to problems such as unbalanced meals and food waste. Therefore, there is a need for a system that can solve these issues and propose more personalized and balanced meals.

[0310] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0311] In this invention, the server includes an image recognition means for identifying ingredients in the user's refrigerator, a means for integrating the identified ingredient information with a user profile, a means for generating menus and recipes based on the ingredient information and the user profile, a means for generating information on presentation and cutting methods related to the menus and recipes, a means for transmitting the generated information to a user terminal, a means for listing ingredients that are in short supply and automatically generating a shopping list, a means for analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, a means for suggesting drink pairings and restaurant menus, and a sentiment analysis means for analyzing the user's emotional state and generating menus and recipes based on the sentiment information. This enables the system to suggest optimal menus and recipes based on the user's emotions and health condition, and to efficiently manage ingredients accordingly.

[0312] "Image recognition means" refers to the technology and algorithms used to analyze images of the inside of the refrigerator taken by the user and identify the ingredients present.

[0313] "User profile" refers to an individual data set that includes a user's preferences, health status, dietary intake information, seasonal information, etc.

[0314] "Means for generating menus and recipes" refers to algorithms and processes for generating appropriate menus and specific recipes based on the user profile and information about ingredients in the refrigerator.

[0315] The term "means for generating information on presentation and cutting methods" refers to a technology that generates specific instructions on how to present and cut ingredients after cooking in relation to the generated menu and recipe.

[0316] "Means for sending information to a user's terminal" refers to the technology for sending various information generated on the server side (menus, recipes, presentation methods, etc.) to a user's terminal such as a smartphone or tablet via the Internet.

[0317] "Shopping list generation means" refers to a technology and process that automatically lists ingredients that are in short supply based on information about ingredients in the refrigerator and the generated recipe, and provides the list to the user.

[0318] "Means for analyzing preferences and nutritional needs" refers to algorithms and technologies for collecting and analyzing the preferences and nutritional needs of each family member and suggesting balanced meals.

[0319] "Means for suggesting drink pairings and restaurant menus" refers to the algorithms and technologies for suggesting drinks that match the generated menu and restaurant menus.

[0320] "Emotion analysis means" refers to the technology and algorithms used to analyze a user's emotional state and generate menus and recipes based on that information.

[0321] This system manages the ingredients in a user's refrigerator and suggests menus and recipes based on that information. By combining it with an emotion engine, it is possible to suggest meals that correspond to the user's emotions. The system is composed of a server and a user terminal.

[0322] System Configuration

[0323] The server is a back-end system that includes an image recognition unit, a database, a recipe generation unit, a plating support unit, a shopping list generation unit, a preference analysis unit, and an emotion engine. The server processes data sent from the user's device and generates the necessary information.

[0324] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server, and corresponds to a smartphone, tablet, etc. An application is installed on the terminal, and the user interacts with the system through this application.

[0325] Program processing overview

[0326] First, the user takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. In doing so, it uses an image recognition algorithm to extract the name and quantity of each ingredient and stores them in a database. Specifically, it uses libraries such as TensorFlow and OpenCV.

[0327] Next, the user inputs their preferences, health status, and dietary information via their device. This information is sent to the server, which updates the user profile. The server then generates optimal menus and recipes based on the user profile and the identified ingredients. The menu generation algorithm is run using Python and R libraries (scikit-learn, pandas).

[0328] The generated menus and recipes include both those that use only available ingredients and those that can be realized by supplementing ingredients that are in short supply. In addition, suggestions are also made on how to present the food after cooking and how to cut the ingredients, and this information is sent to the user's device.

[0329] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[0330] The server analyzes the preferences and nutritional needs of each family member to suggest balanced meals that everyone can enjoy. This information is also sent to the device, allowing users to easily implement the suggested menu. The server also provides information on drink pairings, dining out menus, and recommended restaurants.

[0331] Emotion engine integration

[0332] This system uses an emotion engine to recognize the user's emotions in real time. The emotion engine can detect the user's emotional state through voice input and facial expression recognition. This emotional information is directly reflected in the menu and recipe suggestions generated by the server.

[0333] For example, if a user is feeling down, the emotion engine can detect this information and suggest colorful, nutritious dishes to lift their spirits. On the other hand, if the user is tired, it can suggest recipes that are easy to prepare or that use ingredients that have a relaxing effect. In this way, personalized meal suggestions based on the emotion engine are possible.

[0334] Specific examples

[0335] For example, a user takes a photo of the inside of their refrigerator with their smartphone and uploads it to a server via an application. The server identifies the presence of tomatoes, chicken, and spinach from the image. The user then inputs their preferences, such as "I would like to eat a low-carb diet" and "I like tomato dishes." The emotion engine also detects emotional information from the user's voice input, such as "I feel a little stressed."

[0336] The server uses this information to suggest low-carb recipes, such as tomato chicken stew, that contain ingredients that have stress-reducing properties. The suggestions also include presentation and cutting instructions. If the missing ingredient is cream, the server adds it to the shopping list and notifies the user.

[0337] Users can cook according to the suggested recipes and beautifully arrange the food on a plate using the provided presentation. They can also get information on relaxing herbal teas to pair with their drinks, as well as healthy restaurants nearby.

[0338] Prompt Sentence Examples

[0339] The following prompt statements can be used:

[0340] "There are tomatoes, chicken, and spinach in the refrigerator. The user wants a low-carb diet and prefers tomato dishes. There is also emotional information that the user is feeling a little stressed. Based on this information, please generate menu and recipe suggestions for the user. Also, please create a shopping list for any missing ingredients that are needed."

[0341] In this way, the system of the present invention integrates the user's emotions with the management of ingredients in the refrigerator, enabling the provision of personalized, healthy, and diverse meals.

[0342] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0343] Step 1:

[0344] Users take photos of the inside of the refrigerator with their smartphone and upload them to the server via a dedicated application.

[0345] Input: Image of the inside of the refrigerator (photo file)

[0346] Specific operation: The user takes a photo of the inside of the refrigerator using the camera function of their smartphone, then launches the application, selects the photo they took, and presses the "Send" button.

[0347] Output: Image data of the inside of the refrigerator is sent to the server.

[0348] Step 2:

[0349] The server analyzes the received image data and identifies the ingredients in the refrigerator in real time.

[0350] Input: Image data of the inside of the refrigerator

[0351] Specific operation: The server applies an image recognition algorithm (TensorFlow or OpenCV) to extract the name and quantity of each ingredient, recognize the characteristics of the ingredients, and convert them into a format that can be stored in a database.

[0352] Output: List of identified ingredients (e.g. 3 tomatoes, 500g chicken, etc.)

[0353] Step 3:

[0354] The server stores the identified ingredient information in a database.

[0355] Input: Identified ingredient list

[0356] Specific operation: The server stores the ingredient information in a database such as MySQL or PostgreSQL. For each identified ingredient, it associates the name and quantity and registers them.

[0357] Output: Updated ingredient database

[0358] Step 4:

[0359] Users enter their preferences, health status, and food intake information into the application and send it to the server.

[0360] Input: User preferences, health status, and dietary information

[0361] Specific operation: The user uses the application's input form to enter information such as "low carbohydrate diet," "prefers tomato dishes," and "high blood pressure." Then, the user presses the "Submit" button to send this information to the server.

[0362] Output: User profile information sent

[0363] Step 5:

[0364] The server uses the user profile and identified ingredient information to generate menus and recipes.

[0365] Input: User profile information, identified ingredient list

[0366] How it works: The server uses Python and R libraries (scikit-learn, pandas) to generate menus and recipes that suit the user's preferences and health status. For example, it generates optimal recipes taking into account conditions such as "low carbohydrates" and "stress reduction."

[0367] Output: Generated menu and recipe information

[0368] Step 6:

[0369] The server transmits the generated menu and recipes, as well as related information on serving and cutting methods, to the user terminal.

[0370] Input: Generated menu and recipe information

[0371] What it does: The server packages the generated information and sends it to the user's smartphone, including a recipe for "Tomato Chicken Stup" and detailed instructions on how to serve and cut the chicken.

[0372] Output: Menu and recipe information sent to the user's device

[0373] Step 7:

[0374] The server lists the ingredients that are missing and automatically generates a shopping list.

[0375] Input: Generated recipe information, current ingredient database

[0376] Specific behavior: The server compares the generated recipe with the current ingredient database to identify any missing ingredients, adds them to the list, and generates a purchase link.

[0377] Output: Shopping list

[0378] Step 8:

[0379] The server analyzes the preferences and nutritional needs of each family member and suggests balanced meals.

[0380] Input: Preferences and nutritional needs of the whole family

[0381] Specific operation: The server analyzes the collected family data and generates a menu that is nutritionally balanced and satisfies everyone.

[0382] Output: Balanced family meal plan

[0383] Step 9:

[0384] Servers will suggest drink pairings and dining out menus.

[0385] Input: Generated menu information

[0386] Specific operation: The server runs an algorithm that recommends drinks and restaurant menus that go well with the meal and generates appropriate pairing suggestions.

[0387] Output: Drink pairing and dining menu suggestions

[0388] Step 10:

[0389] The server analyzes the user's emotional state and generates menus and recipes based on the emotional information.

[0390] Input: User's emotional state (obtained from voice input, facial expression recognition, etc.)

[0391] Specific operation: The emotion engine analyzes the acquired voice and facial expression data to identify the user's emotional state. The server then regenerates menus and recipes based on parameters corresponding to the emotion.

[0392] Output: Emotion-based personalized menu and recipe information

[0393] (Application example 2)

[0394] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0395] Conventional menu suggestion systems managed the ingredients in a user's refrigerator and suggested recipes, but they did not adequately provide suggestions based on the user's emotions or instantaneous ingredient management in physical stores. As a result, their effectiveness in improving user satisfaction and reducing food waste was limited. Furthermore, they lacked personalized suggestions that took into account the user's emotions and current preferences, making it difficult to provide the meal suggestions users wanted. Furthermore, they lacked the ability to link home ingredient information when shopping in physical stores, which reduced shopping efficiency and led to wasteful purchases and duplicate ingredient purchases.

[0396] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0397] In this invention, the server includes a means for recognizing the user's emotional state and making meal suggestions based on that emotion, a means for scanning products in a physical store and proposing optimal menus in conjunction with the ingredients in the user's home refrigerator, and a means for integrating a user profile including the user's health condition, preferences, and seasonal information. This enables personalized and appropriate meal suggestions that reflect the user's emotions, health condition, and preferences in real time. Furthermore, menu suggestions can be made in conjunction with the ingredients in the user's home refrigerator even when shopping in a physical store, realizing efficient shopping and waste-free food management.

[0398] "Image recognition means" is a technology that analyzes images captured using a device such as a camera and identifies objects and text contained in the images.

[0399] A "user profile" is a data set that integrates personal information such as a user's preferences, health status, and past dietary history.

[0400] The "means for generating menus and recipes" refers to an algorithm or system that suggests optimal meal menus and cooking methods based on information about ingredients in the user's refrigerator and the user's profile.

[0401] The "means for generating information on presentation and cutting methods" is a technology for generating information to suggest beautiful presentation methods and cutting methods based on the generated recipe.

[0402] The "means for generating a shopping list" is an algorithm or system that detects ingredients that are lacking in the user's refrigerator and automatically creates a shopping list.

[0403] The "means for recognizing emotional states and making meal suggestions according to the emotions" is a system that analyzes the emotions of users from their voices and facial expressions and makes personalized meal suggestions based on that.

[0404] "A means of scanning products in a physical store and combining it with the information on ingredients in the home refrigerator to suggest the optimal menu" is a system that uses a device such as a smartphone to scan products in the store, and then links that information with the information on ingredients in the home refrigerator to suggest the optimal menu.

[0405] "Means for analyzing preferences and nutritional needs and proposing balanced meals" refers to technology that analyzes the preferences and nutritional status of individuals and families, and then proposes nutritionally balanced meals based on that analysis.

[0406] The present invention is a meal recommendation system that integrates a user's emotional state with information about ingredients in the refrigerator and provides shopping support in physical stores. This system is mainly composed of a server and a user terminal. Specific embodiments are as follows.

[0407] System Configuration

[0408] 1. Server

[0409] The server is a back-end system that includes an image recognition means, a database, a recipe generation means, a plating support means, a shopping list generation means, a preference analysis means, and an emotion engine.

[0410] 2. User Device

[0411] The user terminal is a device such as a smartphone or tablet that can be used to take photos of the food in the refrigerator and send them to the server. It also has a function for scanning products in physical stores.

[0412] Program processing

[0413] 1. Image Recognition

[0414] Users take photos of the inside of their refrigerator with their smartphones and upload them to the system. The server analyzes the received images and identifies the ingredients in the refrigerator in real time. It uses an image recognition algorithm (e.g., OpenCV) to extract the name and quantity of each ingredient and stores them in a database.

[0415] 2. User Profile and Preference Analysis

[0416] Users input their preferences, health status, and seasonal information through their terminals. This information is sent to the server, and the user profile is updated. This profile is then analyzed using the preference analysis means.

[0417] 3. Emotion Engine

[0418] The emotion engine detects the user's emotional state through voice input and facial expression recognition. For example, if a user says "I'm feeling a little stressed," this information is analyzed.

[0419] 4. Recipe Generation

[0420] The server generates optimal menus and recipes based on the user profile and the identified ingredient information. Emotional information generated by the emotion engine is also taken into account. The generated menus and recipes include both those that use only available ingredients and those that supplement ingredients that are lacking. The server also suggests ways to present the food after cooking and how to cut the ingredients.

[0421] 5. In-store scanning

[0422] When users shop at a physical store, they scan products with their smartphones. The server analyzes the images and lists any missing ingredients that need to be purchased. It also combines the scanned products with information about ingredients at home to suggest optimal meal plans.

[0423] 6. Shopping List Generation

[0424] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[0425] Specific examples

[0426] For example, if a user wants to use tomatoes and chicken from their refrigerator, they can take a picture of the refrigerator with their smartphone and upload it. They can also say, "I'm feeling a little stressed." Based on this information, the server will suggest low-carb recipes using tomatoes that will help reduce stress. When in a physical store, they can scan the products in the store with the app and easily see what ingredients they need to buy.

[0427] Prompt Sentence Examples

[0428] 1. Prompt (Image Recognition): "The user has uploaded a photo of the inside of their refrigerator. Please recognize the ingredients in this image and store them in a database."

[0429] 2. Prompt (Emotion Recognition): "Please analyze the emotion from the user's voice input. In this case, the input is 'I feel a little stressed.'"

[0430] 3. Prompt (Recipe Generation): "Based on the stored ingredient data and the user's sentiment data, please suggest colorful and nutritious dishes."

[0431] 4. Prompt (in-store): "Scan the products in the store and combine them with the ingredients in your refrigerator at home to suggest the perfect meal plan."

[0432] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0433] Step 1:

[0434] The user takes a photo of the inside of the refrigerator with the device and uploads it to the system. The input is the image of the inside of the refrigerator, and the output is image data sent to the server. At this time, the device uses its camera function to obtain the image data and sends it to the server via the network.

[0435] Step 2:

[0436] The server analyzes the received images and identifies the ingredients in the refrigerator in real time. The input is the uploaded image data, and the output is the identified ingredient information. The server uses an image recognition algorithm (e.g., OpenCV) to analyze the ingredients in the image, extract the names and quantities, and store them in a database.

[0437] Step 3:

[0438] The user inputs their preferences, health status, and seasonal information through the device. The input is the user's personal information, and the output is profile data sent to the server. The device then fills in a form with the data entered by the user and sends it to the server.

[0439] Step 4:

[0440] The server updates the user profile based on the received preference and health information. The input is the user's profile data, and the output is the updated user profile. The server accesses the database and updates the existing profile with the new information.

[0441] Step 5:

[0442] The emotion engine detects the user's emotional state through voice input and facial expression recognition. The input is the user's voice and image data, and the output is the detected emotional information. The server uses voice recognition technology (e.g., Google Cloud Speech-to-Text) and facial expression recognition technology to analyze the user's current emotion and store the information in a database.

[0443] Step 6:

[0444] The server generates optimal menus and recipes based on the updated user profile, identified ingredient information, and emotion information. The inputs are profile data, ingredient data, and emotion data, and the output is the generated menus and recipes. The server integrates this data and uses a recipe generation algorithm to suggest optimal menus.

[0445] Step 7:

[0446] The generated menu and recipes also include information on presentation and cutting methods. The input is the generated menu, and the output is detailed cooking instructions. The server generates detailed information on presentation and cutting methods for the recipe to provide dishes that are easy for users to understand, and sends it to the user's terminal.

[0447] Step 8:

[0448] When a user scans an item in a physical store, the terminal takes a photo of the item and sends it to the server. The input is the product image in the store, and the output is image data sent to the server. The terminal scans the item in the store and sends the scanned image to the server.

[0449] Step 9:

[0450] The server analyzes the received product images, identifies what has been scanned, lists the missing ingredients, and notifies the user. The input is the product image and information about ingredients in the refrigerator, and the output is a list of missing ingredients. The server uses image recognition technology to analyze the scanned product information, compares it with data from the home refrigerator to identify missing ingredients, and automatically generates the list and sends it to the user's device.

[0451] Step 10:

[0452] When missing ingredients are identified, the server automatically generates a shopping list and sends it to the user. The input is information about the missing ingredients, and the output is a shopping list. The server generates a shopping list including a purchase link based on the list of missing ingredients and sends it to the user's terminal.

[0453] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0454] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0455] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0456] [Second embodiment]

[0457] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0458] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0459] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0460] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0461] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0462] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0463] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0464] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0465] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0466] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0467] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0468] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0469] The present invention is a system for managing ingredients stored in a user's refrigerator and proposing menus and recipes based on the ingredients, and is comprised of a server and a user terminal. Specific embodiments of the present invention will be described below.

[0470] System Configuration

[0471] The server is a back-end system that includes image recognition, a database, recipe generation, plating assistance, shopping list generation, and preference analysis. The server processes data sent from the user's device and generates the necessary information.

[0472] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server, and corresponds to a smartphone, tablet, etc. An application is installed on the terminal, and the user interacts with the system through this application.

[0473] Program processing overview

[0474] First, the user takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. At this time, it uses an image recognition algorithm to extract the name and quantity of each ingredient and stores them in a database.

[0475] Next, the user inputs their preferences, health status, and dietary information via their device. This information is sent to the server, which updates the user profile. The server then generates optimal menus and recipes based on the user profile and the identified ingredients.

[0476] The generated menus and recipes include both those that use only available ingredients and those that can be realized by supplementing ingredients that are in short supply. In addition, suggestions are also made on how to present the food after cooking and how to cut the ingredients, and this information is sent to the user's device.

[0477] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[0478] Finally, the server analyzes the preferences and nutritional needs of each family member and suggests balanced meals that everyone can enjoy. This information is also sent to the device, allowing the user to easily implement the suggested menu. The server also provides information on drink pairings, dining out menus, and restaurant recommendations.

[0479] Specific examples

[0480] For example, a user takes a photo of the inside of their refrigerator with their smartphone and uploads it to a server through an application. The server identifies the presence of tomatoes, chicken, and spinach in the image. The user then enters their personal information, such as "I want a low-carb diet" and "I like tomato dishes."

[0481] The server uses this information to suggest low-carb recipes, such as tomato chicken stew, and provides presentation and cutting instructions. If you need more cream than the spinach you need, the server adds it to your shopping list and notifies you.

[0482] Users can follow suggested recipes and beautifully arrange the food on a plate using the provided plating methods. They can also get recommendations for white wine pairings and information on nearby healthy restaurants.

[0483] In this way, the system of the present invention supports the user in planning their meals and provides a healthy and varied diet.

[0484] The processing flow will be explained below.

[0485] Step 1:

[0486] The user takes a photo of the food items in the refrigerator using their smartphone (terminal).

[0487] Specific operation: The user launches the camera app on their smartphone and takes a picture of the ingredients in the refrigerator.

[0488] Step 2:

[0489] The terminal uploads the captured photos to the server.

[0490] Specific operation: The device sends the captured photo data to the specified API endpoint.

[0491] Step 3:

[0492] The server analyzes the received image data and identifies the ingredients.

[0493] Specific operation: The server uses an image recognition algorithm (e.g., Convolutional Neural Network) to identify the type and quantity of ingredients.

[0494] Step 4:

[0495] The server stores the identified ingredient information in a database.

[0496] Specific operation: The extracted list of ingredients and their quantities are associated with the user's profile and recorded in the database.

[0497] Step 5:

[0498] Users input their preferences, health status, and food intake information via the terminal.

[0499] Specific operation: The user enters various information into a form within the application and presses the submit button.

[0500] Step 6:

[0501] The terminal transmits the input user information to the server.

[0502] Specific operation: The device sends the entered preference and health status data to the server via API.

[0503] Step 7:

[0504] The server adds or updates the received user information to the profile.

[0505] Specific operations: The server merges the existing user profile with the newly received information and updates the database.

[0506] Step 8:

[0507] The server integrates the information about ingredients in the refrigerator with the updated user profile to generate menus and recipes.

[0508] What it does: Uses an AI recommendation system to suggest optimal menus and recipes based on user profile and available ingredients.

[0509] Step 9:

[0510] The server sends the generated menu and recipes to the terminal.

[0511] Specific operation: The proposed recipe and its cooking instructions are sent to the device in JSON format via API.

[0512] Step 10:

[0513] The server generates images of how to arrange and cut the food based on the proposed menu.

[0514] Specific operations: Generate presentation guides and visual materials after cooking and prepare them as image data.

[0515] Step 11:

[0516] The server sends the presentation image and guide information to the terminal.

[0517] Specific operation: The generated presentation guide and images are sent to the device via API.

[0518] Step 12:

[0519] The server lists the ingredients that are missing and generates a shopping list.

[0520] Specific behavior: Compare the inventory in the refrigerator with the suggested recipe and make a list of missing ingredients.

[0521] Step 13:

[0522] The server transmits the shopping list to the terminal.

[0523] Specific operation: The list of ingredients that are in short supply and their purchase links are sent to the device via API.

[0524] Step 14:

[0525] The server analyzes the preferences and nutritional needs of each family member and generates a balanced menu.

[0526] Specific operation: The AI ​​references each user profile and generates a menu that can be enjoyed by all.

[0527] Step 15:

[0528] The server sends the proposed balanced menu to the terminal.

[0529] Specific operation: The generated menu and its procedures are sent to the device via API.

[0530] Step 16:

[0531] The server will suggest drink pairings to go with the menu, dining out menus, and recommended restaurants.

[0532] What it does: Recommends drinks and dining options based on a suggested menu and creates a list.

[0533] Step 17:

[0534] The server transmits this additional information to the terminal and notifies the user.

[0535] Specific operation: Recommendation information is sent to the device via API and notified to the user.

[0536] Example 1

[0537] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0538] In today's busy lifestyles, efficiently managing ingredients and planning meals is a difficult task. Properly managing ingredients in the refrigerator and providing menus and recipes that match the user's preferences and health status are important for achieving a healthy and diverse diet. However, conventional systems do not adequately address these needs, forcing users to spend time managing ingredients and searching for recipes. Furthermore, it is difficult to provide suggestions that balance the preferences and nutritional needs of each family member.

[0539] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0540] In this invention, the server includes an image recognition unit for identifying ingredients in the user's refrigerator, a unit for integrating the identified ingredient information with a user profile, a unit using a generative AI model to generate menus and recipes based on the ingredient information and the user profile, a unit using an image generation algorithm to generate information on presentation and cutting methods related to the menus and recipes, a unit for sending the generated information to a user terminal, a unit for listing missing ingredients and automatically generating a shopping list, a unit for analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, and a unit for suggesting drink pairings, restaurant menus, and recommended restaurants. This allows for efficient management of ingredients in the refrigerator and automatically generating optimal menus and recipes tailored to the user's preferences and health status. It also provides balanced meal suggestions and automatic purchasing assistance for missing ingredients.

[0541] "Image recognition means" refers to means that includes algorithms and techniques for identifying ingredients in the refrigerator.

[0542] A "user profile" is a collection of personal information related to a user, such as the user's health status, preferences, and seasonal information.

[0543] A "generative AI model" is an artificial intelligence model that automatically generates menus and recipes based on ingredient information and user profiles.

[0544] An "image generation algorithm" is an algorithm for generating information on presentation and cutting methods related to menus and recipes in the form of realistic images.

[0545] A "user terminal" is a device that a user uses to interact with the system, such as a smartphone or tablet.

[0546] The "shopping list generating means" is a means for listing ingredients that are in short supply and automatically generating a shopping list.

[0547] The "preference analysis tool" is a tool for analyzing the preferences and nutritional needs of each family member and proposing balanced meals.

[0548] The "pairing suggestion means" is a means for suggesting drink pairings, dining out menus, and information on recommended restaurants.

[0549] MODE FOR CARRYING OUT THE INVENTION

[0550] The present invention is a system that manages ingredients stored in a user's refrigerator and suggests menus and recipes based on the ingredients. The system is composed of a server and a user terminal. Specific embodiments of the present invention are described below.

[0551] The server is a backend system that includes image recognition means, a database, recipe generation means, plating assistance means, shopping list generation means, and preference analysis means. The server processes data sent from the user's device and generates the necessary information. For server hardware, a general-purpose server machine (e.g., a server equipped with a high-performance CPU, sufficient memory, and storage) is used. For software, image recognition algorithms such as "YOLO" or "Google Cloud Vision API" are used, databases such as "MySQL," "Firebase," or "AWS DynamoDB" are used, and generative AI models such as "GPT-3" are used.

[0552] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server; it can be a smartphone or tablet. A dedicated application is installed on the terminal, and the user interacts with the system through this application. This application has functions for taking pictures, entering data, and receiving notifications. The terminal hardware is a smartphone with a camera (e.g., Android or iPhone). The software is a dedicated mobile application. This app provides a user interface that allows the user to upload images, enter profile information, and receive notifications.

[0553] The user first takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. To do this, the user opens the refrigerator door and takes a photo of the entire interior with the smartphone camera. Next, the user taps the "Upload" button on the application to send the image. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. For the analysis, it uses the image recognition algorithm "YOLO" and "Google Cloud Vision API." The server then stores the names and quantities of the identified ingredients in a database. This database uses "MySQL," "Firebase," or "AWS DynamoDB."

[0554] Next, the user enters their preferences, health status, and dietary information via their device. For example, if the user prefers a "low-carb" or "low-fat" diet, they enter this information into a form within the app and tap the "Submit" button. The server receives this information and updates the user profile. The server then generates optimal menus and recipes based on the updated user profile and the identified ingredient information. In this process, a generative AI model such as "GPT-3" is used to generate recipes that take into account the user's preferences and available ingredients.

[0555] In addition to the recipe, the server also suggests how to arrange the food after cooking and how to cut the ingredients. To do this, it uses an "image generation AI model" (e.g., DALL-E) to generate a sample image of the presentation and sends it to the user's device. If any ingredients are missing, the server lists them and automatically generates a shopping list. This shopping list also includes a link to purchase the food online and notifies the user.

[0556] The server also analyzes the preferences and nutritional needs of each family member and suggests balanced meals. This analysis could potentially use the Human Nutrition API. It also provides information on drink pairings, dining out menus, and recommended restaurants. Drink pairing information can be obtained from the Wine.com API, while restaurant information can be obtained from the Google Places API or Yelp API.

[0557] Specific examples

[0558] For example, a user can take a photo of the contents of their refrigerator with their smartphone and upload it to a server through an application. The server can identify the presence of tomatoes, chicken, and spinach in the image. The user then enters information into the app, such as "I'm looking for a low-carb diet" and "I like tomato dishes." Based on this information, the server can suggest a low-carb recipe for tomato chicken stew. If the user has the spinach needed for this recipe in their refrigerator but is short on cream, the server will add the cream to their shopping list and notify them. The user can then cook the dish according to the suggested recipe and arrange it on a plate using the suggested plating method. The server can also suggest white wine pairings and provide information about nearby healthy restaurants.

[0559] Prompt example

[0560] "You take a photo of the contents of your refrigerator and upload it to the system. It identifies that you have tomatoes, chicken, and spinach, and requests a low-carb meal and a tomato dish."

[0561] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0562] Program processing flow

[0563] Step 1:

[0564] Users take photos of the inside of the refrigerator with their smartphone (device) and upload them to the server through the application. Specifically, the user opens the refrigerator door and takes a photo of the entire interior with the smartphone camera. Next, they tap the "upload" button in the application to send the image.

[0565] Input: Image of the inside of the refrigerator

[0566] Output: Image data sent to the server

[0567] Step 2:

[0568] The server analyzes the received images and identifies the ingredients in the refrigerator in real time. It uses an image recognition algorithm (e.g., YOLO or Google Cloud Vision API) to extract the names and quantities of ingredients and store them in a database. Specifically, after receiving the images, the server runs the image recognition software and stores the identified ingredient data in MySQL, Firebase, or AWS DynamoDB.

[0569] Input: Image data uploaded by the user

[0570] Output: Database update information including ingredient names and quantities

[0571] Step 3:

[0572] Users input their preferences, health status, and dietary information via their device. Specifically, the user enters information such as "low-carb" or "low-fat" into the application's input form and taps the "Submit" button. The server receives this information and updates the user profile.

[0573] Input: User preferences and health data

[0574] Output: Updated user profile

[0575] Step 4:

[0576] The server generates optimal menus and recipes based on the updated user profile and identified ingredient information. Specifically, the server uses a generative AI model (e.g., GPT-3) to generate recipes that take into account the user's preferences and available ingredients.

[0577] Input: Updated user profile and ingredient information

[0578] Output: Generated menu and recipes

[0579] Step 5:

[0580] The server then suggests post-cooking plating and cutting methods related to the generated menu and recipe. Specifically, the server uses an image generation AI model (e.g., DALL-E) to generate sample plating images and sends this information to the user's device.

[0581] Input: Generated menu and recipe

[0582] Output: Suggestions for presentation and cutting of ingredients

[0583] Step 6:

[0584] The server lists the ingredients that are in short supply and automatically generates a shopping list. Specifically, the server checks for ingredients that are in short supply, creates a shopping list using the Amazon Web Services Lambda function, and notifies the user with a link to purchase the ingredients.

[0585] Input: Ingredient information from the database and the generated recipe

[0586] Output: List of ingredients that are in short supply and a shopping list

[0587] Step 7:

[0588] The server analyzes the preferences and nutritional needs of each family member and suggests balanced meals. Specifically, the server uses the "Human Nutrition API" to calculate nutritional information and compare it with the preference information of each family member. In addition, the server also provides drink pairings and recommended restaurant information.

[0589] Input: Data on the preferences and health status of all family members

[0590] Output: Balanced meal suggestions, drink pairings and restaurant recommendations

[0591] (Application example 1)

[0592] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0593] Conventional food management systems were able to identify ingredients in the refrigerator and suggest recipes based on the user's preferences and health information. However, they were unable to automatically process the entire process, from purchasing missing ingredients to delivery. Furthermore, they lacked optimal recipe suggestions using generative AI models, making it impossible to achieve the high level of personalization desired by users. The present invention aims to solve these issues and provide users with a consistent, personalized meal suggestion and delivery service.

[0594] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0595] In this invention, the server includes an image recognition unit for identifying ingredients in the user's refrigerator, a unit for integrating the identified ingredient information with a user profile, a unit for generating menus and recipes based on the ingredient information and the user profile, a unit for generating information on plating and cutting methods related to the menus and recipes, a unit for sending the generated information to a user terminal, a unit for listing missing ingredients and automatically generating a shopping list, a unit for analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, a unit for suggesting drink pairings and restaurant menus, a unit for automatically selecting missing ingredients based on the identified ingredient information and delivering them together with cooking, and a unit for generating optimal dishes and recipes using a generative AI model. This allows users to manage ingredients in their refrigerator in real time, receive automatic delivery of missing ingredients, and enjoy highly personalized menus and dishes.

[0596] "Image recognition means for identifying ingredients in a user's refrigerator" refers to technology for analyzing images of the inside of a refrigerator taken by a user and identifying ingredients.

[0597] "Means for integrating the identified ingredient information with a user profile" refers to a method for integrating ingredient information identified in a refrigerator with user preferences, health information, etc.

[0598] "Means for generating menus and recipes based on the ingredient information and user profile" refers to technology for generating optimal menus and recipes based on identified ingredient information and user profile information.

[0599] The term "means for generating information on presentation and cutting methods related to the menu and recipe" refers to a method for generating information on presentation and cutting methods for the proposed menu and recipe.

[0600] "Means for transmitting the generated information to the user terminal" refers to a communication method for transmitting information such as the generated menu, recipes, and presentation methods to the user terminal.

[0601] "Means for listing missing ingredients and automatically generating a shopping list" refers to a method for listing missing ingredients required for a proposed recipe and automatically generating a shopping list.

[0602] "Means for analyzing the preferences and nutritional needs of all family members and suggesting balanced meals" refers to a method for analyzing the preferences and nutritional needs of all family members and suggesting balanced meals based on that analysis.

[0603] "Means for suggesting drink pairings and restaurant menus" refers to methods for recommending drinks that go well with the proposed dishes and restaurant menus.

[0604] "Means for automatically picking up missing ingredients based on the identified ingredient information and delivering them together with cooking" refers to a method for automatically picking up missing ingredients based on identified ingredient information and delivering them together with cooking.

[0605] "Means for generating optimal dishes and recipes using a generative AI model" refers to technology that uses a generative AI model to generate optimal dishes and recipes based on the user's preferences and information about ingredients in the refrigerator.

[0606] The present invention relates to a system that manages ingredients in a user's refrigerator and proposes menus and recipes based on the user's preferences and health information. This allows the user to receive personalized meal suggestions, including automatic delivery of ingredients that the user is running low on. Specific embodiments of the present invention are described below.

[0607] System Configuration

[0608] This system mainly consists of a server and a terminal.

[0609] 1. Server:

[0610] The server is a backend system that includes an image recognition means, a database, a recipe generation means, a plating support means, a shopping list generation means, a preference analysis means, and a generative AI model.

[0611] The server has the function of processing data sent from the user's terminal and generating the necessary information.

[0612] 2. Terminal:

[0613] The terminal is a device that allows the user to take a photo of the inside of the refrigerator and send it to the server. This can be a smartphone or tablet.

[0614] Applications are installed on the terminals, and users interact with the system through these applications.

[0615] Program processing overview

[0616] 1. Image Recognition Methods:

[0617] First, the user takes a photo of the inside of the refrigerator using their smartphone (terminal) and uploads it to the system.

[0618] The server analyzes the received images and identifies the ingredients in the refrigerator in real time, using an image recognition algorithm (e.g., TensorFlow) to extract the name and quantity of each ingredient and store them in a database.

[0619] 2. Data integration and recipe generation:

[0620] Users input their preferences, health status, and dietary information through their device, which is then sent to the server and the user profile is updated.

[0621] The server generates optimal menus and recipes based on the user profile and identified ingredient information. It generates dishes and recipes using a generative AI model (e.g., a generative AI model).

[0622] 3. Presentation and Delivery:

[0623] The generated menus and recipes include both those that use only available ingredients and those that can be achieved by supplementing missing ingredients.

[0624] Suggestions are also given on how to arrange the food after cooking and how to cut the ingredients, and this information is sent to the user's terminal.

[0625] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[0626] The server also provides a means to automatically pick up missing ingredients and deliver them along with cooking.

[0627] 4. Meeting the tastes and nutritional needs of the whole family:

[0628] It analyzes the preferences and nutritional needs of each family member and suggests balanced meals that everyone can enjoy. This information is also sent to the device, allowing the user to easily put the suggested menu into practice.

[0629] The server also provides information on drink pairings, dining out menus, and recommended restaurants.

[0630] Specific examples

[0631] For example, a user can take a photo of the contents of their refrigerator with their smartphone and upload it to a server through an application. The server can identify the presence of tomatoes, chicken, and spinach in the image. The user then inputs their personal information, such as "I'm looking for a low-carb diet" and "I like tomato dishes." The server then uses this information to suggest low-carb recipes, such as tomato chicken stew. It also provides an image of how the food will look after cooking and how to cut it. If the user needs more cream than the spinach they need, the server adds it to their shopping list and notifies them. The user can then prepare the dish according to the suggested recipe and beautifully arrange it on a plate using the provided plating method. They can also receive recommendations for white wines to pair with their drinks and information about nearby healthy restaurants.

[0632] Prompt Sentence Examples

[0633] The following prompt sentence is used as input to the generative AI model:

[0634] I have tomatoes and chicken in my fridge. Analyze images of the fridge and suggest low-calorie meals using these ingredients. If the suggested meal contains missing ingredients, add them to the shopping list and offer them as a delivery option.

[0635] The above is an embodiment of the present invention, which allows users to efficiently manage ingredients in their refrigerators and receive personalized meal suggestions and delivery services.

[0636] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0637] Step 1:

[0638] The user takes a photo of the inside of the refrigerator using their smartphone (device). The image is saved on the device and becomes the input data for the system.

[0639] Step 2:

[0640] The user uploads a photo of the inside of the refrigerator to the server via the terminal. The terminal sends the image data to the server via the Internet. In this case, the image file becomes the input data.

[0641] Step 3:

[0642] The server analyzes the received images and identifies the ingredients in the refrigerator in real time. Specifically, the server uses an image recognition algorithm (e.g., TensorFlow) to extract the name and quantity of each ingredient and store the identified ingredient information in a database. In this process, image data is the input and identified ingredient information is the output.

[0643] Step 4:

[0644] Users input their preferences, health status, and dietary information through their device. This information is sent from the device to the server, where the user profile is updated. Here, preferences and health information are input data, and the updated user profile is output.

[0645] Step 5:

[0646] The server generates optimal menus and recipes based on the user profile and identified ingredient information. The server uses a generative AI model (e.g., generative AI model) to generate dishes and recipes based on the prompt. In this process, the user profile and ingredient information are input data, and the generated menus and recipes are output.

[0647] Step 6:

[0648] The server generates information on how to arrange and cut ingredients related to the menu and recipe, allowing the user to receive specific cooking and presentation advice. In this process, the menu and recipe are input data, and information on how to arrange and cut ingredients is output.

[0649] Step 7:

[0650] The server sends the generated information to the user's terminal, where the user can view the proposed menu, recipes, presentation methods, and other information. In this process, all the information generated up to that point is input data, and the data sent to the user's terminal is the output.

[0651] Step 8:

[0652] The server lists the missing ingredients and automatically generates a shopping list. Furthermore, this list also includes a purchase link so that the user can easily purchase the missing ingredients. In this process, the required ingredient information from the menu and recipes is the input data, and the shopping list is the output.

[0653] Step 9:

[0654] The server automatically picks up missing ingredients based on the identified ingredient information, cooks them, and delivers them. Specifically, an order is issued to a partner delivery service. In this process, the missing ingredient information is the input data, and a delivery order is the output.

[0655] Step 10:

[0656] The server analyzes the preferences and nutritional needs of each family member and suggests balanced meals. It also suggests drink pairings and restaurant menus, and users can receive this information on their devices. In this process, the input data is the profiles of each family member, and the output is balanced meals and pairing information.

[0657] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0658] This system manages the ingredients in a user's refrigerator and suggests menus and recipes based on that information. In particular, by combining it with an emotion engine, this system makes it possible to suggest meals based on the user's emotions. It is composed of a server and a user terminal.

[0659] System Configuration

[0660] The server is a back-end system that includes image recognition, a database, recipe generation, plating assistance, shopping list generation, preference analysis, and an emotion engine. The server processes data sent from the user's device and generates the necessary information.

[0661] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server, and corresponds to a smartphone, tablet, etc. An application is installed on the terminal, and the user interacts with the system through this application.

[0662] Program processing overview

[0663] First, the user takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. At this time, it uses an image recognition algorithm to extract the name and quantity of each ingredient and stores them in a database.

[0664] Next, the user inputs their preferences, health status, and dietary information via their device. This information is sent to the server, which updates the user profile. The server then generates optimal menus and recipes based on the user profile and the identified ingredients.

[0665] The generated menus and recipes include both those that use only available ingredients and those that can be realized by supplementing ingredients that are in short supply. In addition, suggestions are also made on how to present the food after cooking and how to cut the ingredients, and this information is sent to the user's device.

[0666] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[0667] The server analyzes the preferences and nutritional needs of each family member to suggest balanced meals that everyone can enjoy. This information is also sent to the device, allowing users to easily implement the suggested menu. The server also provides information on drink pairings, dining out menus, and recommended restaurants.

[0668] Emotion engine integration

[0669] This system uses an emotion engine to recognize the user's emotions in real time. The emotion engine can detect the user's emotional state through voice input and facial expression recognition. This emotional information is directly reflected in the menu and recipe suggestions generated by the server.

[0670] For example, if a user is feeling down, the emotion engine can detect this information and suggest colorful, nutritious dishes to lift their spirits. On the other hand, if the user is tired, it can suggest recipes that are easy to prepare or that use ingredients that have a relaxing effect. In this way, personalized meal suggestions based on the emotion engine are possible.

[0671] Specific examples

[0672] For example, a user can take a photo of the inside of their refrigerator with their smartphone and upload it to a server through an application, which can then identify the presence of tomatoes, chicken, and spinach in the image.

[0673] Next, the user inputs their preferences, such as "I want a low-carb diet" and "I like tomato dishes." The emotion engine also detects emotional information from the user's voice input, such as "I'm feeling a little stressed."

[0674] The server uses this information to suggest low-carb recipes, such as tomato chicken stew, that contain ingredients that have stress-reducing properties. The suggestions also include presentation and cutting instructions. If cream is the missing ingredient, the server adds it to the shopping list and notifies the user.

[0675] Users can cook according to the suggested recipes and beautifully arrange the food on a plate using the provided plating methods. They can also get information on relaxing herbal teas to pair with their drinks, as well as healthy restaurants nearby.

[0676] In this way, the system of the present invention integrates the user's emotions with the management of ingredients in the refrigerator, enabling the provision of personalized, healthy, and diverse meals.

[0677] The processing flow will be explained below.

[0678] Step 1:

[0679] The user takes a photo of the food items in the refrigerator using their smartphone (terminal).

[0680] Specific operation: The user launches the camera app on their smartphone and takes a picture of the ingredients in the refrigerator.

[0681] Step 2:

[0682] The terminal uploads the captured photos to the server.

[0683] Specific operation: The device sends the captured photo data to the specified API endpoint.

[0684] Step 3:

[0685] The server analyzes the received image data and identifies the ingredients.

[0686] Specific operation: The server uses an image recognition algorithm (e.g., Convolutional Neural Network) to identify the type and quantity of ingredients.

[0687] Step 4:

[0688] The server stores the identified ingredient information in a database.

[0689] Specific operation: The extracted list of ingredients and their quantities are associated with the user's profile and recorded in the database.

[0690] Step 5:

[0691] Users input their preferences, health status, and food intake information via the terminal.

[0692] Specific operation: The user enters various information into a form within the application and presses the submit button.

[0693] Step 6:

[0694] The terminal transmits the input user information to the server.

[0695] Specific operation: The device sends the entered preference and health status data to the server via API.

[0696] Step 7:

[0697] The server adds or updates the received user information to the profile.

[0698] Specific operations: The server merges the existing user profile with the newly received information and updates the database.

[0699] Step 8:

[0700] The device collects voice input and facial expression images to detect the user's emotions.

[0701] Specific operation: The terminal records the user's voice or saves a photograph of the user's face.

[0702] Step 9:

[0703] The terminal uses an emotion engine to analyze the user's emotions.

[0704] Specific behavior: Estimates the user's emotional state using voice and facial expression analysis algorithms.

[0705] Step 10:

[0706] The terminal transmits the analysis results to the server.

[0707] Specific operation: The estimated emotion information is sent to the server via API.

[0708] Step 11:

[0709] The server integrates information about ingredients in the refrigerator with the updated user profile and emotional information to generate menus and recipes.

[0710] What it does: Uses an AI recommendation system to suggest optimal menus and recipes based on user profile, available ingredients, and emotional information.

[0711] Step 12:

[0712] The server sends the generated menu and recipes to the terminal.

[0713] Specific operation: The proposed recipe and its cooking instructions are sent to the device in JSON format via API.

[0714] Step 13:

[0715] The server generates images of how to arrange and cut the food based on the proposed menu.

[0716] Specific operations: Generate presentation guides and visual materials after cooking and prepare them as image data.

[0717] Step 14:

[0718] The server sends the presentation image and guide information to the terminal.

[0719] Specific operation: The generated presentation guide and images are sent to the device via API.

[0720] Step 15:

[0721] The server lists the ingredients that are missing and generates a shopping list.

[0722] Specific behavior: Compare the inventory in the refrigerator with the suggested recipe and make a list of missing ingredients.

[0723] Step 16:

[0724] The server transmits the shopping list to the terminal.

[0725] Specific operation: The list of ingredients that are in short supply and their purchase links are sent to the device via API.

[0726] Step 17:

[0727] The server analyzes the preferences and nutritional needs of each family member and generates a balanced menu.

[0728] Specific operation: The AI ​​references each user profile and generates a menu that can be enjoyed by all.

[0729] Step 18:

[0730] The server sends the proposed balanced menu to the terminal.

[0731] Specific operation: The generated menu and its procedures are sent to the device via API.

[0732] Step 19:

[0733] The server will suggest drink pairings to go with the menu, dining out menus, and recommended restaurants.

[0734] What it does: Recommends drinks and dining options based on a suggested menu and creates a list.

[0735] Step 20:

[0736] The server transmits this additional information to the terminal and notifies the user.

[0737] Specific operation: Recommendation information is sent to the device via API and notified to the user.

[0738] Example 2

[0739] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0740] In modern life, there is a demand for efficient management of ingredients in the refrigerator and for optimal menus and recipes to be proposed based on the user's preferences and health status. However, conventional systems have limitations in ingredient management and menu proposals, and are unable to realize personalized meal proposals that reflect the user's emotional state. Furthermore, there are insufficient means for grasping ingredient inventory status and efficiently replenishing ingredients that are in short supply. This leads to problems such as unbalanced meals and food waste. Therefore, there is a need for a system that can solve these issues and propose more personalized and balanced meals.

[0741] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0742] In this invention, the server includes an image recognition means for identifying ingredients in the user's refrigerator, a means for integrating the identified ingredient information with a user profile, a means for generating menus and recipes based on the ingredient information and the user profile, a means for generating information on presentation and cutting methods related to the menus and recipes, a means for transmitting the generated information to a user terminal, a means for listing ingredients that are in short supply and automatically generating a shopping list, a means for analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, a means for suggesting drink pairings and restaurant menus, and a sentiment analysis means for analyzing the user's emotional state and generating menus and recipes based on the sentiment information. This enables the system to suggest optimal menus and recipes based on the user's emotions and health condition, and to efficiently manage ingredients accordingly.

[0743] "Image recognition means" refers to the technology and algorithms used to analyze images of the inside of the refrigerator taken by the user and identify the ingredients present.

[0744] "User profile" refers to an individual data set that includes a user's preferences, health status, dietary intake information, seasonal information, etc.

[0745] "Means for generating menus and recipes" refers to algorithms and processes for generating appropriate menus and specific recipes based on the user profile and information about ingredients in the refrigerator.

[0746] The term "means for generating information on presentation and cutting methods" refers to a technology that generates specific instructions on how to present and cut ingredients after cooking in relation to the generated menu and recipe.

[0747] "Means for sending information to a user's terminal" refers to the technology for sending various information generated on the server side (menus, recipes, presentation methods, etc.) to a user's terminal such as a smartphone or tablet via the Internet.

[0748] "Shopping list generation means" refers to a technology and process that automatically lists ingredients that are in short supply based on information about ingredients in the refrigerator and the generated recipe, and provides the list to the user.

[0749] "Means for analyzing preferences and nutritional needs" refers to algorithms and technologies for collecting and analyzing the preferences and nutritional needs of each family member and suggesting balanced meals.

[0750] "Means for suggesting drink pairings and restaurant menus" refers to the algorithms and technologies for suggesting drinks that match the generated menu and restaurant menus.

[0751] "Emotion analysis means" refers to the technology and algorithms used to analyze a user's emotional state and generate menus and recipes based on that information.

[0752] This system manages the ingredients in a user's refrigerator and suggests menus and recipes based on that information. By combining it with an emotion engine, it is possible to suggest meals that correspond to the user's emotions. The system is composed of a server and a user terminal.

[0753] System Configuration

[0754] The server is a back-end system that includes an image recognition unit, a database, a recipe generation unit, a plating support unit, a shopping list generation unit, a preference analysis unit, and an emotion engine. The server processes data sent from the user's device and generates the necessary information.

[0755] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server, and corresponds to a smartphone, tablet, etc. An application is installed on the terminal, and the user interacts with the system through this application.

[0756] Program processing overview

[0757] First, the user takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. In doing so, it uses an image recognition algorithm to extract the name and quantity of each ingredient and stores them in a database. Specifically, it uses libraries such as TensorFlow and OpenCV.

[0758] Next, the user inputs their preferences, health status, and dietary information via their device. This information is sent to the server, which updates the user profile. The server then generates optimal menus and recipes based on the user profile and the identified ingredients. The menu generation algorithm is run using Python and R libraries (scikit-learn, pandas).

[0759] The generated menus and recipes include both those that use only available ingredients and those that can be realized by supplementing ingredients that are in short supply. In addition, suggestions are also made on how to present the food after cooking and how to cut the ingredients, and this information is sent to the user's device.

[0760] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[0761] The server analyzes the preferences and nutritional needs of each family member to suggest balanced meals that everyone can enjoy. This information is also sent to the device, allowing users to easily implement the suggested menu. The server also provides information on drink pairings, dining out menus, and recommended restaurants.

[0762] Emotion engine integration

[0763] This system uses an emotion engine to recognize the user's emotions in real time. The emotion engine can detect the user's emotional state through voice input and facial expression recognition. This emotional information is directly reflected in the menu and recipe suggestions generated by the server.

[0764] For example, if a user is feeling down, the emotion engine can detect this information and suggest colorful, nutritious dishes to lift their spirits. On the other hand, if the user is tired, it can suggest recipes that are easy to prepare or that use ingredients that have a relaxing effect. In this way, personalized meal suggestions based on the emotion engine are possible.

[0765] Specific examples

[0766] For example, a user takes a photo of the inside of their refrigerator with their smartphone and uploads it to a server via an application. The server identifies the presence of tomatoes, chicken, and spinach from the image. The user then inputs their preferences, such as "I would like to eat a low-carb diet" and "I like tomato dishes." The emotion engine also detects emotional information from the user's voice input, such as "I feel a little stressed."

[0767] The server uses this information to suggest low-carb recipes, such as tomato chicken stew, that contain ingredients that have stress-reducing properties. The suggestions also include presentation and cutting instructions. If the missing ingredient is cream, the server adds it to the shopping list and notifies the user.

[0768] Users can cook according to the suggested recipes and beautifully arrange the food on a plate using the provided presentation. They can also get information on relaxing herbal teas to pair with their drinks, as well as healthy restaurants nearby.

[0769] Prompt Sentence Examples

[0770] The following prompt statements can be used:

[0771] "There are tomatoes, chicken, and spinach in the refrigerator. The user wants a low-carb diet and prefers tomato dishes. There is also emotional information that the user is feeling a little stressed. Based on this information, please generate menu and recipe suggestions for the user. Also, please create a shopping list for any missing ingredients that are needed."

[0772] In this way, the system of the present invention integrates the user's emotions with the management of ingredients in the refrigerator, enabling the provision of personalized, healthy, and diverse meals.

[0773] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0774] Step 1:

[0775] Users take photos of the inside of the refrigerator with their smartphone and upload them to the server via a dedicated application.

[0776] Input: Image of the inside of the refrigerator (photo file)

[0777] Specific operation: The user takes a photo of the inside of the refrigerator using the camera function of their smartphone, then launches the application, selects the photo they took, and presses the "Send" button.

[0778] Output: Image data of the inside of the refrigerator is sent to the server.

[0779] Step 2:

[0780] The server analyzes the received image data and identifies the ingredients in the refrigerator in real time.

[0781] Input: Image data of the inside of the refrigerator

[0782] Specific operation: The server applies an image recognition algorithm (TensorFlow or OpenCV) to extract the name and quantity of each ingredient, recognize the characteristics of the ingredients, and convert them into a format that can be stored in a database.

[0783] Output: List of identified ingredients (e.g. 3 tomatoes, 500g chicken, etc.)

[0784] Step 3:

[0785] The server stores the identified ingredient information in a database.

[0786] Input: Identified ingredient list

[0787] Specific operation: The server stores the ingredient information in a database such as MySQL or PostgreSQL. For each identified ingredient, it associates the name and quantity and registers them.

[0788] Output: Updated ingredient database

[0789] Step 4:

[0790] Users enter their preferences, health status, and food intake information into the application and send it to the server.

[0791] Input: User preferences, health status, and dietary information

[0792] Specific operation: The user uses the application's input form to enter information such as "low carbohydrate diet," "prefers tomato dishes," and "high blood pressure." Then, the user presses the "Submit" button to send this information to the server.

[0793] Output: User profile information sent

[0794] Step 5:

[0795] The server uses the user profile and identified ingredient information to generate menus and recipes.

[0796] Input: User profile information, identified ingredient list

[0797] How it works: The server uses Python and R libraries (scikit-learn, pandas) to generate menus and recipes that suit the user's preferences and health status. For example, it generates optimal recipes taking into account conditions such as "low carbohydrates" and "stress reduction."

[0798] Output: Generated menu and recipe information

[0799] Step 6:

[0800] The server transmits the generated menu and recipes, as well as related information on serving and cutting methods, to the user terminal.

[0801] Input: Generated menu and recipe information

[0802] What it does: The server packages the generated information and sends it to the user's smartphone, including a recipe for "Tomato Chicken Stup" and detailed instructions on how to serve and cut the chicken.

[0803] Output: Menu and recipe information sent to the user's device

[0804] Step 7:

[0805] The server lists the ingredients that are missing and automatically generates a shopping list.

[0806] Input: Generated recipe information, current ingredient database

[0807] Specific behavior: The server compares the generated recipe with the current ingredient database to identify any missing ingredients, adds them to the list, and generates a purchase link.

[0808] Output: Shopping list

[0809] Step 8:

[0810] The server analyzes the preferences and nutritional needs of each family member and suggests balanced meals.

[0811] Input: Preferences and nutritional needs of the whole family

[0812] Specific operation: The server analyzes the collected family data and generates a menu that is nutritionally balanced and satisfies everyone.

[0813] Output: Balanced family meal plan

[0814] Step 9:

[0815] Servers will suggest drink pairings and dining out menus.

[0816] Input: Generated menu information

[0817] Specific operation: The server runs an algorithm that recommends drinks and restaurant menus that go well with the meal and generates appropriate pairing suggestions.

[0818] Output: Drink pairing and dining menu suggestions

[0819] Step 10:

[0820] The server analyzes the user's emotional state and generates menus and recipes based on the emotional information.

[0821] Input: User's emotional state (obtained from voice input, facial expression recognition, etc.)

[0822] Specific operation: The emotion engine analyzes the acquired voice and facial expression data to identify the user's emotional state. The server then regenerates menus and recipes based on parameters corresponding to the emotion.

[0823] Output: Emotion-based personalized menu and recipe information

[0824] (Application example 2)

[0825] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0826] Conventional menu suggestion systems managed the ingredients in a user's refrigerator and suggested recipes, but they did not adequately provide suggestions based on the user's emotions or instantaneous ingredient management in physical stores. As a result, their effectiveness in improving user satisfaction and reducing food waste was limited. Furthermore, they lacked personalized suggestions that took into account the user's emotions and current preferences, making it difficult to provide the meal suggestions users wanted. Furthermore, they lacked the ability to link home ingredient information when shopping in physical stores, which reduced shopping efficiency and led to wasteful purchases and duplicate ingredient purchases.

[0827] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0828] In this invention, the server includes a means for recognizing the user's emotional state and making meal suggestions based on that emotion, a means for scanning products in a physical store and proposing optimal menus in conjunction with the ingredients in the user's home refrigerator, and a means for integrating a user profile including the user's health condition, preferences, and seasonal information. This enables personalized and appropriate meal suggestions that reflect the user's emotions, health condition, and preferences in real time. Furthermore, menu suggestions can be made in conjunction with the ingredients in the user's home refrigerator even when shopping in a physical store, realizing efficient shopping and waste-free food management.

[0829] "Image recognition means" is a technology that analyzes images captured using a device such as a camera and identifies objects and text contained in the images.

[0830] A "user profile" is a data set that integrates personal information such as a user's preferences, health status, and past dietary history.

[0831] The "means for generating menus and recipes" refers to an algorithm or system that suggests optimal meal menus and cooking methods based on information about ingredients in the user's refrigerator and the user's profile.

[0832] The "means for generating information on presentation and cutting methods" is a technology for generating information to suggest beautiful presentation methods and cutting methods based on the generated recipe.

[0833] The "means for generating a shopping list" is an algorithm or system that detects ingredients that are lacking in the user's refrigerator and automatically creates a shopping list.

[0834] The "means for recognizing emotional states and making meal suggestions according to the emotions" is a system that analyzes the emotions of users from their voices and facial expressions and makes personalized meal suggestions based on that.

[0835] "A means of scanning products in a physical store and combining it with the information on ingredients in the home refrigerator to suggest the optimal menu" is a system that uses a device such as a smartphone to scan products in the store, and then links that information with the information on ingredients in the home refrigerator to suggest the optimal menu.

[0836] "Means for analyzing preferences and nutritional needs and proposing balanced meals" refers to technology that analyzes the preferences and nutritional status of individuals and families, and then proposes nutritionally balanced meals based on that analysis.

[0837] The present invention is a meal recommendation system that integrates a user's emotional state with information about ingredients in the refrigerator and provides shopping support in physical stores. This system is mainly composed of a server and a user terminal. Specific embodiments are as follows.

[0838] System Configuration

[0839] 1. Server

[0840] The server is a back-end system that includes an image recognition means, a database, a recipe generation means, a plating support means, a shopping list generation means, a preference analysis means, and an emotion engine.

[0841] 2. User Device

[0842] The user terminal is a device such as a smartphone or tablet that can be used to take photos of the food in the refrigerator and send them to the server. It also has a function for scanning products in physical stores.

[0843] Program processing

[0844] 1. Image Recognition

[0845] Users take photos of the inside of their refrigerator with their smartphones and upload them to the system. The server analyzes the received images and identifies the ingredients in the refrigerator in real time. It uses an image recognition algorithm (e.g., OpenCV) to extract the name and quantity of each ingredient and stores them in a database.

[0846] 2. User Profile and Preference Analysis

[0847] Users input their preferences, health status, and seasonal information through their terminals. This information is sent to the server, and the user profile is updated. This profile is then analyzed using the preference analysis means.

[0848] 3. Emotion Engine

[0849] The emotion engine detects the user's emotional state through voice input and facial expression recognition. For example, if a user says "I'm feeling a little stressed," this information is analyzed.

[0850] 4. Recipe Generation

[0851] The server generates optimal menus and recipes based on the user profile and the identified ingredient information. Emotional information generated by the emotion engine is also taken into account. The generated menus and recipes include both those that use only available ingredients and those that supplement ingredients that are lacking. The server also suggests ways to present the food after cooking and how to cut the ingredients.

[0852] 5. In-store scanning

[0853] When users shop at a physical store, they scan products with their smartphones. The server analyzes the images and lists any missing ingredients that need to be purchased. It also combines the scanned products with information about ingredients at home to suggest optimal meal plans.

[0854] 6. Shopping List Generation

[0855] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[0856] Specific examples

[0857] For example, if a user wants to use tomatoes and chicken from their refrigerator, they can take a picture of the refrigerator with their smartphone and upload it. They can also say, "I'm feeling a little stressed." Based on this information, the server will suggest low-carb recipes using tomatoes that will help reduce stress. When in a physical store, they can scan the products in the store with the app and easily see what ingredients they need to buy.

[0858] Prompt Sentence Examples

[0859] 1. Prompt (Image Recognition): "The user has uploaded a photo of the inside of their refrigerator. Please recognize the ingredients in this image and store them in a database."

[0860] 2. Prompt (Emotion Recognition): "Please analyze the emotion from the user's voice input. In this case, the input is 'I feel a little stressed.'"

[0861] 3. Prompt (Recipe Generation): "Based on the stored ingredient data and the user's sentiment data, please suggest colorful and nutritious dishes."

[0862] 4. Prompt (in-store): "Scan the products in the store and combine them with the ingredients in your refrigerator at home to suggest the perfect meal plan."

[0863] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0864] Step 1:

[0865] The user takes a photo of the inside of the refrigerator with the device and uploads it to the system. The input is the image of the inside of the refrigerator, and the output is image data sent to the server. At this time, the device uses its camera function to obtain the image data and sends it to the server via the network.

[0866] Step 2:

[0867] The server analyzes the received images and identifies the ingredients in the refrigerator in real time. The input is the uploaded image data, and the output is the identified ingredient information. The server uses an image recognition algorithm (e.g., OpenCV) to analyze the ingredients in the image, extract the names and quantities, and store them in a database.

[0868] Step 3:

[0869] The user inputs their preferences, health status, and seasonal information through the device. The input is the user's personal information, and the output is profile data sent to the server. The device then fills in a form with the data entered by the user and sends it to the server.

[0870] Step 4:

[0871] The server updates the user profile based on the received preference and health information. The input is the user's profile data, and the output is the updated user profile. The server accesses the database and updates the existing profile with the new information.

[0872] Step 5:

[0873] The emotion engine detects the user's emotional state through voice input and facial expression recognition. The input is the user's voice and image data, and the output is the detected emotional information. The server uses voice recognition technology (e.g., Google Cloud Speech-to-Text) and facial expression recognition technology to analyze the user's current emotion and store the information in a database.

[0874] Step 6:

[0875] The server generates optimal menus and recipes based on the updated user profile, identified ingredient information, and emotion information. The inputs are profile data, ingredient data, and emotion data, and the output is the generated menus and recipes. The server integrates this data and uses a recipe generation algorithm to suggest optimal menus.

[0876] Step 7:

[0877] The generated menu and recipes also include information on presentation and cutting methods. The input is the generated menu, and the output is detailed cooking instructions. The server generates detailed information on presentation and cutting methods for the recipe to provide dishes that are easy for users to understand, and sends it to the user's terminal.

[0878] Step 8:

[0879] When a user scans an item in a physical store, the terminal takes a photo of the item and sends it to the server. The input is the product image in the store, and the output is image data sent to the server. The terminal scans the item in the store and sends the scanned image to the server.

[0880] Step 9:

[0881] The server analyzes the received product images, identifies what has been scanned, lists the missing ingredients, and notifies the user. The input is the product image and information about ingredients in the refrigerator, and the output is a list of missing ingredients. The server uses image recognition technology to analyze the scanned product information, compares it with data from the home refrigerator to identify missing ingredients, and automatically generates the list and sends it to the user's device.

[0882] Step 10:

[0883] When missing ingredients are identified, the server automatically generates a shopping list and sends it to the user. The input is information about the missing ingredients, and the output is a shopping list. The server generates a shopping list including a purchase link based on the list of missing ingredients and sends it to the user's terminal.

[0884] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0885] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0886] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0887] [Third embodiment]

[0888] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0889] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0890] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0891] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0892] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0893] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0894] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0895] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0896] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0897] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0898] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0899] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0900] The present invention is a system for managing ingredients stored in a user's refrigerator and proposing menus and recipes based on the ingredients, and is comprised of a server and a user terminal. Specific embodiments of the present invention will be described below.

[0901] System Configuration

[0902] The server is a back-end system that includes image recognition, a database, recipe generation, plating assistance, shopping list generation, and preference analysis. The server processes data sent from the user's device and generates the necessary information.

[0903] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server, and corresponds to a smartphone, tablet, etc. An application is installed on the terminal, and the user interacts with the system through this application.

[0904] Program processing overview

[0905] First, the user takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. At this time, it uses an image recognition algorithm to extract the name and quantity of each ingredient and stores them in a database.

[0906] Next, the user inputs their preferences, health status, and dietary information via their device. This information is sent to the server, which updates the user profile. The server then generates optimal menus and recipes based on the user profile and the identified ingredients.

[0907] The generated menus and recipes include both those that use only available ingredients and those that can be realized by supplementing ingredients that are in short supply. In addition, suggestions are also made on how to present the food after cooking and how to cut the ingredients, and this information is sent to the user's device.

[0908] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[0909] Finally, the server analyzes the preferences and nutritional needs of each family member and suggests balanced meals that everyone can enjoy. This information is also sent to the device, allowing the user to easily implement the suggested menu. The server also provides information on drink pairings, dining out menus, and restaurant recommendations.

[0910] Specific examples

[0911] For example, a user takes a photo of the inside of their refrigerator with their smartphone and uploads it to a server through an application. The server identifies the presence of tomatoes, chicken, and spinach in the image. The user then enters their personal information, such as "I want a low-carb diet" and "I like tomato dishes."

[0912] The server uses this information to suggest low-carb recipes, such as tomato chicken stew, and provides presentation and cutting instructions. If you need more cream than the spinach you need, the server adds it to your shopping list and notifies you.

[0913] Users can follow suggested recipes and beautifully arrange the food on a plate using the provided plating methods. They can also get recommendations for white wine pairings and information on nearby healthy restaurants.

[0914] In this way, the system of the present invention supports the user in planning their meals and provides a healthy and varied diet.

[0915] The processing flow will be explained below.

[0916] Step 1:

[0917] The user takes a photo of the food items in the refrigerator using their smartphone (terminal).

[0918] Specific operation: The user launches the camera app on their smartphone and takes a picture of the ingredients in the refrigerator.

[0919] Step 2:

[0920] The terminal uploads the captured photos to the server.

[0921] Specific operation: The device sends the captured photo data to the specified API endpoint.

[0922] Step 3:

[0923] The server analyzes the received image data and identifies the ingredients.

[0924] Specific operation: The server uses an image recognition algorithm (e.g., Convolutional Neural Network) to identify the type and quantity of ingredients.

[0925] Step 4:

[0926] The server stores the identified ingredient information in a database.

[0927] Specific operation: The extracted list of ingredients and their quantities are associated with the user's profile and recorded in the database.

[0928] Step 5:

[0929] Users input their preferences, health status, and food intake information via the terminal.

[0930] Specific operation: The user enters various information into a form within the application and presses the submit button.

[0931] Step 6:

[0932] The terminal transmits the input user information to the server.

[0933] Specific operation: The device sends the entered preference and health status data to the server via API.

[0934] Step 7:

[0935] The server adds or updates the received user information to the profile.

[0936] Specific operations: The server merges the existing user profile with the newly received information and updates the database.

[0937] Step 8:

[0938] The server integrates the information about ingredients in the refrigerator with the updated user profile to generate menus and recipes.

[0939] What it does: Uses an AI recommendation system to suggest optimal menus and recipes based on user profile and available ingredients.

[0940] Step 9:

[0941] The server sends the generated menu and recipes to the terminal.

[0942] Specific operation: The proposed recipe and its cooking instructions are sent to the device in JSON format via API.

[0943] Step 10:

[0944] The server generates images of how to arrange and cut the food based on the proposed menu.

[0945] Specific operations: Generate presentation guides and visual materials after cooking and prepare them as image data.

[0946] Step 11:

[0947] The server sends the presentation image and guide information to the terminal.

[0948] Specific operation: The generated presentation guide and images are sent to the device via API.

[0949] Step 12:

[0950] The server lists the ingredients that are missing and generates a shopping list.

[0951] Specific behavior: Compare the inventory in the refrigerator with the suggested recipe and make a list of missing ingredients.

[0952] Step 13:

[0953] The server transmits the shopping list to the terminal.

[0954] Specific operation: The list of ingredients that are in short supply and their purchase links are sent to the device via API.

[0955] Step 14:

[0956] The server analyzes the preferences and nutritional needs of each family member and generates a balanced menu.

[0957] Specific operation: The AI ​​references each user profile and generates a menu that can be enjoyed by all.

[0958] Step 15:

[0959] The server sends the proposed balanced menu to the terminal.

[0960] Specific operation: The generated menu and its procedures are sent to the device via API.

[0961] Step 16:

[0962] The server will suggest drink pairings to go with the menu, dining out menus, and recommended restaurants.

[0963] What it does: Recommends drinks and dining options based on a suggested menu and creates a list.

[0964] Step 17:

[0965] The server transmits this additional information to the terminal and notifies the user.

[0966] Specific operation: Recommendation information is sent to the device via API and notified to the user.

[0967] Example 1

[0968] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0969] In today's busy lifestyles, efficiently managing ingredients and planning meals is a difficult task. Properly managing ingredients in the refrigerator and providing menus and recipes that match the user's preferences and health status are important for achieving a healthy and diverse diet. However, conventional systems do not adequately address these needs, forcing users to spend time managing ingredients and searching for recipes. Furthermore, it is difficult to provide suggestions that balance the preferences and nutritional needs of each family member.

[0970] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0971] In this invention, the server includes an image recognition unit for identifying ingredients in the user's refrigerator, a unit for integrating the identified ingredient information with a user profile, a unit using a generative AI model to generate menus and recipes based on the ingredient information and the user profile, a unit using an image generation algorithm to generate information on presentation and cutting methods related to the menus and recipes, a unit for sending the generated information to a user terminal, a unit for listing missing ingredients and automatically generating a shopping list, a unit for analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, and a unit for suggesting drink pairings, restaurant menus, and recommended restaurants. This allows for efficient management of ingredients in the refrigerator and automatically generating optimal menus and recipes tailored to the user's preferences and health status. It also provides balanced meal suggestions and automatic purchasing assistance for missing ingredients.

[0972] "Image recognition means" refers to means that includes algorithms and techniques for identifying ingredients in the refrigerator.

[0973] A "user profile" is a collection of personal information related to a user, such as the user's health status, preferences, and seasonal information.

[0974] A "generative AI model" is an artificial intelligence model that automatically generates menus and recipes based on ingredient information and user profiles.

[0975] An "image generation algorithm" is an algorithm for generating information on presentation and cutting methods related to menus and recipes in the form of realistic images.

[0976] A "user terminal" is a device that a user uses to interact with the system, such as a smartphone or tablet.

[0977] The "shopping list generating means" is a means for listing ingredients that are in short supply and automatically generating a shopping list.

[0978] The "preference analysis tool" is a tool for analyzing the preferences and nutritional needs of each family member and proposing balanced meals.

[0979] The "pairing suggestion means" is a means for suggesting drink pairings, dining out menus, and information on recommended restaurants.

[0980] MODE FOR CARRYING OUT THE INVENTION

[0981] The present invention is a system that manages ingredients stored in a user's refrigerator and suggests menus and recipes based on the ingredients. The system is composed of a server and a user terminal. Specific embodiments of the present invention are described below.

[0982] The server is a backend system that includes image recognition means, a database, recipe generation means, plating assistance means, shopping list generation means, and preference analysis means. The server processes data sent from the user's device and generates the necessary information. For server hardware, a general-purpose server machine (e.g., a server equipped with a high-performance CPU, sufficient memory, and storage) is used. For software, image recognition algorithms such as "YOLO" or "Google Cloud Vision API" are used, databases such as "MySQL," "Firebase," or "AWS DynamoDB" are used, and generative AI models such as "GPT-3" are used.

[0983] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server; it can be a smartphone or tablet. A dedicated application is installed on the terminal, and the user interacts with the system through this application. This application has functions for taking pictures, entering data, and receiving notifications. The terminal hardware is a smartphone with a camera (e.g., Android or iPhone). The software is a dedicated mobile application. This app provides a user interface that allows the user to upload images, enter profile information, and receive notifications.

[0984] The user first takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. To do this, the user opens the refrigerator door and takes a photo of the entire interior with the smartphone camera. Next, the user taps the "Upload" button on the application to send the image. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. For the analysis, it uses the image recognition algorithm "YOLO" and "Google Cloud Vision API." The server then stores the names and quantities of the identified ingredients in a database. This database uses "MySQL," "Firebase," or "AWS DynamoDB."

[0985] Next, the user enters their preferences, health status, and dietary information via their device. For example, if the user prefers a "low-carb" or "low-fat" diet, they enter this information into a form within the app and tap the "Submit" button. The server receives this information and updates the user profile. The server then generates optimal menus and recipes based on the updated user profile and the identified ingredient information. In this process, a generative AI model such as "GPT-3" is used to generate recipes that take into account the user's preferences and available ingredients.

[0986] In addition to the recipe, the server also suggests how to arrange the food after cooking and how to cut the ingredients. To do this, it uses an "image generation AI model" (e.g., DALL-E) to generate a sample image of the presentation and sends it to the user's device. If any ingredients are missing, the server lists them and automatically generates a shopping list. This shopping list also includes a link to purchase the food online and notifies the user.

[0987] The server also analyzes the preferences and nutritional needs of each family member and suggests balanced meals. This analysis could potentially use the Human Nutrition API. It also provides information on drink pairings, dining out menus, and recommended restaurants. Drink pairing information can be obtained from the Wine.com API, while restaurant information can be obtained from the Google Places API or Yelp API.

[0988] Specific examples

[0989] For example, a user can take a photo of the contents of their refrigerator with their smartphone and upload it to a server through an application. The server can identify the presence of tomatoes, chicken, and spinach in the image. The user then enters information into the app, such as "I'm looking for a low-carb diet" and "I like tomato dishes." Based on this information, the server can suggest a low-carb recipe for tomato chicken stew. If the user has the spinach needed for this recipe in their refrigerator but is short on cream, the server will add the cream to their shopping list and notify them. The user can then cook the dish according to the suggested recipe and arrange it on a plate using the suggested plating method. The server can also suggest white wine pairings and provide information about nearby healthy restaurants.

[0990] Prompt example

[0991] "You take a photo of the contents of your refrigerator and upload it to the system. It identifies that you have tomatoes, chicken, and spinach, and requests a low-carb meal and a tomato dish."

[0992] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0993] Program processing flow

[0994] Step 1:

[0995] Users take photos of the inside of the refrigerator with their smartphone (device) and upload them to the server through the application. Specifically, the user opens the refrigerator door and takes a photo of the entire interior with the smartphone camera. Next, they tap the "upload" button in the application to send the image.

[0996] Input: Image of the inside of the refrigerator

[0997] Output: Image data sent to the server

[0998] Step 2:

[0999] The server analyzes the received images and identifies the ingredients in the refrigerator in real time. It uses an image recognition algorithm (e.g., YOLO or Google Cloud Vision API) to extract the names and quantities of ingredients and store them in a database. Specifically, after receiving the images, the server runs the image recognition software and stores the identified ingredient data in MySQL, Firebase, or AWS DynamoDB.

[1000] Input: Image data uploaded by the user

[1001] Output: Database update information including ingredient names and quantities

[1002] Step 3:

[1003] Users input their preferences, health status, and dietary information via their device. Specifically, the user enters information such as "low-carb" or "low-fat" into the application's input form and taps the "Submit" button. The server receives this information and updates the user profile.

[1004] Input: User preferences and health data

[1005] Output: Updated user profile

[1006] Step 4:

[1007] The server generates optimal menus and recipes based on the updated user profile and identified ingredient information. Specifically, the server uses a generative AI model (e.g., GPT-3) to generate recipes that take into account the user's preferences and available ingredients.

[1008] Input: Updated user profile and ingredient information

[1009] Output: Generated menu and recipes

[1010] Step 5:

[1011] The server then suggests post-cooking plating and cutting methods related to the generated menu and recipe. Specifically, the server uses an image generation AI model (e.g., DALL-E) to generate sample plating images and sends this information to the user's device.

[1012] Input: Generated menu and recipe

[1013] Output: Suggestions for presentation and cutting of ingredients

[1014] Step 6:

[1015] The server lists the ingredients that are in short supply and automatically generates a shopping list. Specifically, the server checks for ingredients that are in short supply, creates a shopping list using the Amazon Web Services Lambda function, and notifies the user with a link to purchase the ingredients.

[1016] Input: Ingredient information from the database and the generated recipe

[1017] Output: List of ingredients that are in short supply and a shopping list

[1018] Step 7:

[1019] The server analyzes the preferences and nutritional needs of each family member and suggests balanced meals. Specifically, the server uses the "Human Nutrition API" to calculate nutritional information and compare it with the preference information of each family member. In addition, the server also provides drink pairings and recommended restaurant information.

[1020] Input: Data on the preferences and health status of all family members

[1021] Output: Balanced meal suggestions, drink pairings and restaurant recommendations

[1022] (Application example 1)

[1023] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1024] Conventional food management systems were able to identify ingredients in the refrigerator and suggest recipes based on the user's preferences and health information. However, they were unable to automatically process the entire process, from purchasing missing ingredients to delivery. Furthermore, they lacked optimal recipe suggestions using generative AI models, making it impossible to achieve the high level of personalization desired by users. The present invention aims to solve these issues and provide users with a consistent, personalized meal suggestion and delivery service.

[1025] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1026] In this invention, the server includes an image recognition unit for identifying ingredients in the user's refrigerator, a unit for integrating the identified ingredient information with a user profile, a unit for generating menus and recipes based on the ingredient information and the user profile, a unit for generating information on plating and cutting methods related to the menus and recipes, a unit for sending the generated information to a user terminal, a unit for listing missing ingredients and automatically generating a shopping list, a unit for analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, a unit for suggesting drink pairings and restaurant menus, a unit for automatically selecting missing ingredients based on the identified ingredient information and delivering them together with cooking, and a unit for generating optimal dishes and recipes using a generative AI model. This allows users to manage ingredients in their refrigerator in real time, receive automatic delivery of missing ingredients, and enjoy highly personalized menus and dishes.

[1027] "Image recognition means for identifying ingredients in a user's refrigerator" refers to technology for analyzing images of the inside of a refrigerator taken by a user and identifying ingredients.

[1028] "Means for integrating the identified ingredient information with a user profile" refers to a method for integrating ingredient information identified in a refrigerator with user preferences, health information, etc.

[1029] "Means for generating menus and recipes based on the ingredient information and user profile" refers to technology for generating optimal menus and recipes based on identified ingredient information and user profile information.

[1030] The term "means for generating information on presentation and cutting methods related to the menu and recipe" refers to a method for generating information on presentation and cutting methods for the proposed menu and recipe.

[1031] "Means for transmitting the generated information to the user terminal" refers to a communication method for transmitting information such as the generated menu, recipes, and presentation methods to the user terminal.

[1032] "Means for listing missing ingredients and automatically generating a shopping list" refers to a method for listing missing ingredients required for a proposed recipe and automatically generating a shopping list.

[1033] "Means for analyzing the preferences and nutritional needs of all family members and suggesting balanced meals" refers to a method for analyzing the preferences and nutritional needs of all family members and suggesting balanced meals based on that analysis.

[1034] "Means for suggesting drink pairings and restaurant menus" refers to methods for recommending drinks that go well with the proposed dishes and restaurant menus.

[1035] "Means for automatically picking up missing ingredients based on the identified ingredient information and delivering them together with cooking" refers to a method for automatically picking up missing ingredients based on identified ingredient information and delivering them together with cooking.

[1036] "Means for generating optimal dishes and recipes using a generative AI model" refers to technology that uses a generative AI model to generate optimal dishes and recipes based on the user's preferences and information about ingredients in the refrigerator.

[1037] The present invention relates to a system that manages ingredients in a user's refrigerator and proposes menus and recipes based on the user's preferences and health information. This allows the user to receive personalized meal suggestions, including automatic delivery of ingredients that the user is running low on. Specific embodiments of the present invention are described below.

[1038] System Configuration

[1039] This system mainly consists of a server and a terminal.

[1040] 1. Server:

[1041] The server is a backend system that includes an image recognition means, a database, a recipe generation means, a plating support means, a shopping list generation means, a preference analysis means, and a generative AI model.

[1042] The server has the function of processing data sent from the user's terminal and generating the necessary information.

[1043] 2. Terminal:

[1044] The terminal is a device that allows the user to take a photo of the inside of the refrigerator and send it to the server. This can be a smartphone or tablet.

[1045] Applications are installed on the terminals, and users interact with the system through these applications.

[1046] Program processing overview

[1047] 1. Image Recognition Methods:

[1048] First, the user takes a photo of the inside of the refrigerator using their smartphone (terminal) and uploads it to the system.

[1049] The server analyzes the received images and identifies the ingredients in the refrigerator in real time, using an image recognition algorithm (e.g., TensorFlow) to extract the name and quantity of each ingredient and store them in a database.

[1050] 2. Data integration and recipe generation:

[1051] Users input their preferences, health status, and dietary information through their device, which is then sent to the server and the user profile is updated.

[1052] The server generates optimal menus and recipes based on the user profile and identified ingredient information. It generates dishes and recipes using a generative AI model (e.g., a generative AI model).

[1053] 3. Presentation and Delivery:

[1054] The generated menus and recipes include both those that use only available ingredients and those that can be achieved by supplementing missing ingredients.

[1055] Suggestions are also given on how to arrange the food after cooking and how to cut the ingredients, and this information is sent to the user's terminal.

[1056] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[1057] The server also provides a means to automatically pick up missing ingredients and deliver them along with cooking.

[1058] 4. Meeting the tastes and nutritional needs of the whole family:

[1059] It analyzes the preferences and nutritional needs of each family member and suggests balanced meals that everyone can enjoy. This information is also sent to the device, allowing the user to easily put the suggested menu into practice.

[1060] The server also provides information on drink pairings, dining out menus, and recommended restaurants.

[1061] Specific examples

[1062] For example, a user can take a photo of the contents of their refrigerator with their smartphone and upload it to a server through an application. The server can identify the presence of tomatoes, chicken, and spinach in the image. The user then inputs their personal information, such as "I'm looking for a low-carb diet" and "I like tomato dishes." The server then uses this information to suggest low-carb recipes, such as tomato chicken stew. It also provides an image of how the food will look after cooking and how to cut it. If the user needs more cream than the spinach they need, the server adds it to their shopping list and notifies them. The user can then prepare the dish according to the suggested recipe and beautifully arrange it on a plate using the provided plating method. They can also receive recommendations for white wines to pair with their drinks and information about nearby healthy restaurants.

[1063] Prompt Sentence Examples

[1064] The following prompt sentence is used as input to the generative AI model:

[1065] I have tomatoes and chicken in my fridge. Analyze images of the fridge and suggest low-calorie meals using these ingredients. If the suggested meal contains missing ingredients, add them to the shopping list and offer them as a delivery option.

[1066] The above is an embodiment of the present invention, which allows users to efficiently manage ingredients in their refrigerators and receive personalized meal suggestions and delivery services.

[1067] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1068] Step 1:

[1069] The user takes a photo of the inside of the refrigerator using their smartphone (device). The image is saved on the device and becomes the input data for the system.

[1070] Step 2:

[1071] The user uploads a photo of the inside of the refrigerator to the server via the terminal. The terminal sends the image data to the server via the Internet. In this case, the image file becomes the input data.

[1072] Step 3:

[1073] The server analyzes the received images and identifies the ingredients in the refrigerator in real time. Specifically, the server uses an image recognition algorithm (e.g., TensorFlow) to extract the name and quantity of each ingredient and store the identified ingredient information in a database. In this process, image data is the input and identified ingredient information is the output.

[1074] Step 4:

[1075] Users input their preferences, health status, and dietary information through their device. This information is sent from the device to the server, where the user profile is updated. Here, preferences and health information are input data, and the updated user profile is output.

[1076] Step 5:

[1077] The server generates optimal menus and recipes based on the user profile and identified ingredient information. The server uses a generative AI model (e.g., generative AI model) to generate dishes and recipes based on the prompt. In this process, the user profile and ingredient information are input data, and the generated menus and recipes are output.

[1078] Step 6:

[1079] The server generates information on how to arrange and cut ingredients related to the menu and recipe, allowing the user to receive specific cooking and presentation advice. In this process, the menu and recipe are input data, and information on how to arrange and cut ingredients is output.

[1080] Step 7:

[1081] The server sends the generated information to the user's terminal, where the user can view the proposed menu, recipes, presentation methods, and other information. In this process, all the information generated up to that point is input data, and the data sent to the user's terminal is the output.

[1082] Step 8:

[1083] The server lists the missing ingredients and automatically generates a shopping list. Furthermore, this list also includes a purchase link so that the user can easily purchase the missing ingredients. In this process, the required ingredient information from the menu and recipes is the input data, and the shopping list is the output.

[1084] Step 9:

[1085] The server automatically picks up missing ingredients based on the identified ingredient information, cooks them, and delivers them. Specifically, an order is issued to a partner delivery service. In this process, the missing ingredient information is the input data, and a delivery order is the output.

[1086] Step 10:

[1087] The server analyzes the preferences and nutritional needs of each family member and suggests balanced meals. It also suggests drink pairings and restaurant menus, and users can receive this information on their devices. In this process, the input data is the profiles of each family member, and the output is balanced meals and pairing information.

[1088] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1089] This system manages the ingredients in a user's refrigerator and suggests menus and recipes based on that information. In particular, by combining it with an emotion engine, this system makes it possible to suggest meals based on the user's emotions. It is composed of a server and a user terminal.

[1090] System Configuration

[1091] The server is a back-end system that includes image recognition, a database, recipe generation, plating assistance, shopping list generation, preference analysis, and an emotion engine. The server processes data sent from the user's device and generates the necessary information.

[1092] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server, and corresponds to a smartphone, tablet, etc. An application is installed on the terminal, and the user interacts with the system through this application.

[1093] Program processing overview

[1094] First, the user takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. At this time, it uses an image recognition algorithm to extract the name and quantity of each ingredient and stores them in a database.

[1095] Next, the user inputs their preferences, health status, and dietary information via their device. This information is sent to the server, which updates the user profile. The server then generates optimal menus and recipes based on the user profile and the identified ingredients.

[1096] The generated menus and recipes include both those that use only available ingredients and those that can be realized by supplementing ingredients that are in short supply. In addition, suggestions are also made on how to present the food after cooking and how to cut the ingredients, and this information is sent to the user's device.

[1097] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[1098] The server analyzes the preferences and nutritional needs of each family member to suggest balanced meals that everyone can enjoy. This information is also sent to the device, allowing users to easily implement the suggested menu. The server also provides information on drink pairings, dining out menus, and recommended restaurants.

[1099] Emotion engine integration

[1100] This system uses an emotion engine to recognize the user's emotions in real time. The emotion engine can detect the user's emotional state through voice input and facial expression recognition. This emotional information is directly reflected in the menu and recipe suggestions generated by the server.

[1101] For example, if a user is feeling down, the emotion engine can detect this information and suggest colorful, nutritious dishes to lift their spirits. On the other hand, if the user is tired, it can suggest recipes that are easy to prepare or that use ingredients that have a relaxing effect. In this way, personalized meal suggestions based on the emotion engine are possible.

[1102] Specific examples

[1103] For example, a user can take a photo of the inside of their refrigerator with their smartphone and upload it to a server through an application, which can then identify the presence of tomatoes, chicken, and spinach in the image.

[1104] Next, the user inputs their preferences, such as "I want a low-carb diet" and "I like tomato dishes." The emotion engine also detects emotional information from the user's voice input, such as "I'm feeling a little stressed."

[1105] The server uses this information to suggest low-carb recipes, such as tomato chicken stew, that contain ingredients that have stress-reducing properties. The suggestions also include presentation and cutting instructions. If cream is the missing ingredient, the server adds it to the shopping list and notifies the user.

[1106] Users can cook according to the suggested recipes and beautifully arrange the food on a plate using the provided plating methods. They can also get information on relaxing herbal teas to pair with their drinks, as well as healthy restaurants nearby.

[1107] In this way, the system of the present invention integrates the user's emotions with the management of ingredients in the refrigerator, enabling the provision of personalized, healthy, and diverse meals.

[1108] The processing flow will be explained below.

[1109] Step 1:

[1110] The user takes a photo of the food items in the refrigerator using their smartphone (terminal).

[1111] Specific operation: The user launches the camera app on their smartphone and takes a picture of the ingredients in the refrigerator.

[1112] Step 2:

[1113] The terminal uploads the captured photos to the server.

[1114] Specific operation: The device sends the captured photo data to the specified API endpoint.

[1115] Step 3:

[1116] The server analyzes the received image data and identifies the ingredients.

[1117] Specific operation: The server uses an image recognition algorithm (e.g., Convolutional Neural Network) to identify the type and quantity of ingredients.

[1118] Step 4:

[1119] The server stores the identified ingredient information in a database.

[1120] Specific operation: The extracted list of ingredients and their quantities are associated with the user's profile and recorded in the database.

[1121] Step 5:

[1122] Users input their preferences, health status, and food intake information via the terminal.

[1123] Specific operation: The user enters various information into a form within the application and presses the submit button.

[1124] Step 6:

[1125] The terminal transmits the input user information to the server.

[1126] Specific operation: The device sends the entered preference and health status data to the server via API.

[1127] Step 7:

[1128] The server adds or updates the received user information to the profile.

[1129] Specific operations: The server merges the existing user profile with the newly received information and updates the database.

[1130] Step 8:

[1131] The device collects voice input and facial expression images to detect the user's emotions.

[1132] Specific operation: The terminal records the user's voice or saves a photograph of the user's face.

[1133] Step 9:

[1134] The terminal uses an emotion engine to analyze the user's emotions.

[1135] Specific behavior: Estimates the user's emotional state using voice and facial expression analysis algorithms.

[1136] Step 10:

[1137] The terminal transmits the analysis results to the server.

[1138] Specific operation: The estimated emotion information is sent to the server via API.

[1139] Step 11:

[1140] The server integrates information about ingredients in the refrigerator with the updated user profile and emotional information to generate menus and recipes.

[1141] What it does: Uses an AI recommendation system to suggest optimal menus and recipes based on user profile, available ingredients, and emotional information.

[1142] Step 12:

[1143] The server sends the generated menu and recipes to the terminal.

[1144] Specific operation: The proposed recipe and its cooking instructions are sent to the device in JSON format via API.

[1145] Step 13:

[1146] The server generates images of how to arrange and cut the food based on the proposed menu.

[1147] Specific operations: Generate presentation guides and visual materials after cooking and prepare them as image data.

[1148] Step 14:

[1149] The server sends the presentation image and guide information to the terminal.

[1150] Specific operation: The generated presentation guide and images are sent to the device via API.

[1151] Step 15:

[1152] The server lists the ingredients that are missing and generates a shopping list.

[1153] Specific behavior: Compare the inventory in the refrigerator with the suggested recipe and make a list of missing ingredients.

[1154] Step 16:

[1155] The server transmits the shopping list to the terminal.

[1156] Specific operation: The list of ingredients that are in short supply and their purchase links are sent to the device via API.

[1157] Step 17:

[1158] The server analyzes the preferences and nutritional needs of each family member and generates a balanced menu.

[1159] Specific operation: The AI ​​references each user profile and generates a menu that can be enjoyed by all.

[1160] Step 18:

[1161] The server sends the proposed balanced menu to the terminal.

[1162] Specific operation: The generated menu and its procedures are sent to the device via API.

[1163] Step 19:

[1164] The server will suggest drink pairings to go with the menu, dining out menus, and recommended restaurants.

[1165] What it does: Recommends drinks and dining options based on a suggested menu and creates a list.

[1166] Step 20:

[1167] The server transmits this additional information to the terminal and notifies the user.

[1168] Specific operation: Recommendation information is sent to the device via API and notified to the user.

[1169] Example 2

[1170] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1171] In modern life, there is a demand for efficient management of ingredients in the refrigerator and for optimal menus and recipes to be proposed based on the user's preferences and health status. However, conventional systems have limitations in ingredient management and menu proposals, and are unable to realize personalized meal proposals that reflect the user's emotional state. Furthermore, there are insufficient means for grasping ingredient inventory status and efficiently replenishing ingredients that are in short supply. This leads to problems such as unbalanced meals and food waste. Therefore, there is a need for a system that can solve these issues and propose more personalized and balanced meals.

[1172] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1173] In this invention, the server includes an image recognition means for identifying ingredients in the user's refrigerator, a means for integrating the identified ingredient information with a user profile, a means for generating menus and recipes based on the ingredient information and the user profile, a means for generating information on presentation and cutting methods related to the menus and recipes, a means for transmitting the generated information to a user terminal, a means for listing ingredients that are in short supply and automatically generating a shopping list, a means for analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, a means for suggesting drink pairings and restaurant menus, and a sentiment analysis means for analyzing the user's emotional state and generating menus and recipes based on the sentiment information. This enables the system to suggest optimal menus and recipes based on the user's emotions and health condition, and to efficiently manage ingredients accordingly.

[1174] "Image recognition means" refers to the technology and algorithms used to analyze images of the inside of the refrigerator taken by the user and identify the ingredients present.

[1175] "User profile" refers to an individual data set that includes a user's preferences, health status, dietary intake information, seasonal information, etc.

[1176] "Means for generating menus and recipes" refers to algorithms and processes for generating appropriate menus and specific recipes based on the user profile and information about ingredients in the refrigerator.

[1177] The term "means for generating information on presentation and cutting methods" refers to a technology that generates specific instructions on how to present and cut ingredients after cooking in relation to the generated menu and recipe.

[1178] "Means for sending information to a user's terminal" refers to the technology for sending various information generated on the server side (menus, recipes, presentation methods, etc.) to a user's terminal such as a smartphone or tablet via the Internet.

[1179] "Shopping list generation means" refers to a technology and process that automatically lists ingredients that are in short supply based on information about ingredients in the refrigerator and the generated recipe, and provides the list to the user.

[1180] "Means for analyzing preferences and nutritional needs" refers to algorithms and technologies for collecting and analyzing the preferences and nutritional needs of each family member and suggesting balanced meals.

[1181] "Means for suggesting drink pairings and restaurant menus" refers to the algorithms and technologies for suggesting drinks that match the generated menu and restaurant menus.

[1182] "Emotion analysis means" refers to the technology and algorithms used to analyze a user's emotional state and generate menus and recipes based on that information.

[1183] This system manages the ingredients in a user's refrigerator and suggests menus and recipes based on that information. By combining it with an emotion engine, it is possible to suggest meals that correspond to the user's emotions. The system is composed of a server and a user terminal.

[1184] System Configuration

[1185] The server is a back-end system that includes an image recognition unit, a database, a recipe generation unit, a plating support unit, a shopping list generation unit, a preference analysis unit, and an emotion engine. The server processes data sent from the user's device and generates the necessary information.

[1186] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server, and corresponds to a smartphone, tablet, etc. An application is installed on the terminal, and the user interacts with the system through this application.

[1187] Program processing overview

[1188] First, the user takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. In doing so, it uses an image recognition algorithm to extract the name and quantity of each ingredient and stores them in a database. Specifically, it uses libraries such as TensorFlow and OpenCV.

[1189] Next, the user inputs their preferences, health status, and dietary information via their device. This information is sent to the server, which updates the user profile. The server then generates optimal menus and recipes based on the user profile and the identified ingredients. The menu generation algorithm is run using Python and R libraries (scikit-learn, pandas).

[1190] The generated menus and recipes include both those that use only available ingredients and those that can be realized by supplementing ingredients that are in short supply. In addition, suggestions are also made on how to present the food after cooking and how to cut the ingredients, and this information is sent to the user's device.

[1191] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[1192] The server analyzes the preferences and nutritional needs of each family member to suggest balanced meals that everyone can enjoy. This information is also sent to the device, allowing users to easily implement the suggested menu. The server also provides information on drink pairings, dining out menus, and recommended restaurants.

[1193] Emotion engine integration

[1194] This system uses an emotion engine to recognize the user's emotions in real time. The emotion engine can detect the user's emotional state through voice input and facial expression recognition. This emotional information is directly reflected in the menu and recipe suggestions generated by the server.

[1195] For example, if a user is feeling down, the emotion engine can detect this information and suggest colorful, nutritious dishes to lift their spirits. On the other hand, if the user is tired, it can suggest recipes that are easy to prepare or that use ingredients that have a relaxing effect. In this way, personalized meal suggestions based on the emotion engine are possible.

[1196] Specific examples

[1197] For example, a user takes a photo of the inside of their refrigerator with their smartphone and uploads it to a server via an application. The server identifies the presence of tomatoes, chicken, and spinach from the image. The user then inputs their preferences, such as "I would like to eat a low-carb diet" and "I like tomato dishes." The emotion engine also detects emotional information from the user's voice input, such as "I feel a little stressed."

[1198] The server uses this information to suggest low-carb recipes, such as tomato chicken stew, that contain ingredients that have stress-reducing properties. The suggestions also include presentation and cutting instructions. If the missing ingredient is cream, the server adds it to the shopping list and notifies the user.

[1199] Users can cook according to the suggested recipes and beautifully arrange the food on a plate using the provided presentation. They can also get information on relaxing herbal teas to pair with their drinks, as well as healthy restaurants nearby.

[1200] Prompt Sentence Examples

[1201] The following prompt statements can be used:

[1202] "There are tomatoes, chicken, and spinach in the refrigerator. The user wants a low-carb diet and prefers tomato dishes. There is also emotional information that the user is feeling a little stressed. Based on this information, please generate menu and recipe suggestions for the user. Also, please create a shopping list for any missing ingredients that are needed."

[1203] In this way, the system of the present invention integrates the user's emotions with the management of ingredients in the refrigerator, enabling the provision of personalized, healthy, and diverse meals.

[1204] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1205] Step 1:

[1206] Users take photos of the inside of the refrigerator with their smartphone and upload them to the server via a dedicated application.

[1207] Input: Image of the inside of the refrigerator (photo file)

[1208] Specific operation: The user takes a photo of the inside of the refrigerator using the camera function of their smartphone, then launches the application, selects the photo they took, and presses the "Send" button.

[1209] Output: Image data of the inside of the refrigerator is sent to the server.

[1210] Step 2:

[1211] The server analyzes the received image data and identifies the ingredients in the refrigerator in real time.

[1212] Input: Image data of the inside of the refrigerator

[1213] Specific operation: The server applies an image recognition algorithm (TensorFlow or OpenCV) to extract the name and quantity of each ingredient, recognize the characteristics of the ingredients, and convert them into a format that can be stored in a database.

[1214] Output: List of identified ingredients (e.g. 3 tomatoes, 500g chicken, etc.)

[1215] Step 3:

[1216] The server stores the identified ingredient information in a database.

[1217] Input: Identified ingredient list

[1218] Specific operation: The server stores the ingredient information in a database such as MySQL or PostgreSQL. For each identified ingredient, it associates the name and quantity and registers them.

[1219] Output: Updated ingredient database

[1220] Step 4:

[1221] Users enter their preferences, health status, and food intake information into the application and send it to the server.

[1222] Input: User preferences, health status, and dietary information

[1223] Specific operation: The user uses the application's input form to enter information such as "low carbohydrate diet," "prefers tomato dishes," and "high blood pressure." Then, the user presses the "Submit" button to send this information to the server.

[1224] Output: User profile information sent

[1225] Step 5:

[1226] The server uses the user profile and identified ingredient information to generate menus and recipes.

[1227] Input: User profile information, identified ingredient list

[1228] How it works: The server uses Python and R libraries (scikit-learn, pandas) to generate menus and recipes that suit the user's preferences and health status. For example, it generates optimal recipes taking into account conditions such as "low carbohydrates" and "stress reduction."

[1229] Output: Generated menu and recipe information

[1230] Step 6:

[1231] The server transmits the generated menu and recipes, as well as related information on serving and cutting methods, to the user terminal.

[1232] Input: Generated menu and recipe information

[1233] What it does: The server packages the generated information and sends it to the user's smartphone, including a recipe for "Tomato Chicken Stup" and detailed instructions on how to serve and cut the chicken.

[1234] Output: Menu and recipe information sent to the user's device

[1235] Step 7:

[1236] The server lists the ingredients that are missing and automatically generates a shopping list.

[1237] Input: Generated recipe information, current ingredient database

[1238] Specific behavior: The server compares the generated recipe with the current ingredient database to identify any missing ingredients, adds them to the list, and generates a purchase link.

[1239] Output: Shopping list

[1240] Step 8:

[1241] The server analyzes the preferences and nutritional needs of each family member and suggests balanced meals.

[1242] Input: Preferences and nutritional needs of the whole family

[1243] Specific operation: The server analyzes the collected family data and generates a menu that is nutritionally balanced and satisfies everyone.

[1244] Output: Balanced family meal plan

[1245] Step 9:

[1246] Servers will suggest drink pairings and dining out menus.

[1247] Input: Generated menu information

[1248] Specific operation: The server runs an algorithm that recommends drinks and restaurant menus that go well with the meal and generates appropriate pairing suggestions.

[1249] Output: Drink pairing and dining menu suggestions

[1250] Step 10:

[1251] The server analyzes the user's emotional state and generates menus and recipes based on the emotional information.

[1252] Input: User's emotional state (obtained from voice input, facial expression recognition, etc.)

[1253] Specific operation: The emotion engine analyzes the acquired voice and facial expression data to identify the user's emotional state. The server then regenerates menus and recipes based on parameters corresponding to the emotion.

[1254] Output: Emotion-based personalized menu and recipe information

[1255] (Application example 2)

[1256] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1257] Conventional menu suggestion systems managed the ingredients in a user's refrigerator and suggested recipes, but they did not adequately provide suggestions based on the user's emotions or instantaneous ingredient management in physical stores. As a result, their effectiveness in improving user satisfaction and reducing food waste was limited. Furthermore, they lacked personalized suggestions that took into account the user's emotions and current preferences, making it difficult to provide the meal suggestions users wanted. Furthermore, they lacked the ability to link home ingredient information when shopping in physical stores, which reduced shopping efficiency and led to wasteful purchases and duplicate ingredient purchases.

[1258] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1259] In this invention, the server includes a means for recognizing the user's emotional state and making meal suggestions based on that emotion, a means for scanning products in a physical store and proposing optimal menus in conjunction with the ingredients in the user's home refrigerator, and a means for integrating a user profile including the user's health condition, preferences, and seasonal information. This enables personalized and appropriate meal suggestions that reflect the user's emotions, health condition, and preferences in real time. Furthermore, menu suggestions can be made in conjunction with the ingredients in the user's home refrigerator even when shopping in a physical store, realizing efficient shopping and waste-free food management.

[1260] "Image recognition means" is a technology that analyzes images captured using a device such as a camera and identifies objects and text contained in the images.

[1261] A "user profile" is a data set that integrates personal information such as a user's preferences, health status, and past dietary history.

[1262] The "means for generating menus and recipes" refers to an algorithm or system that suggests optimal meal menus and cooking methods based on information about ingredients in the user's refrigerator and the user's profile.

[1263] The "means for generating information on presentation and cutting methods" is a technology for generating information to suggest beautiful presentation methods and cutting methods based on the generated recipe.

[1264] The "means for generating a shopping list" is an algorithm or system that detects ingredients that are lacking in the user's refrigerator and automatically creates a shopping list.

[1265] The "means for recognizing emotional states and making meal suggestions according to the emotions" is a system that analyzes the emotions of users from their voices and facial expressions and makes personalized meal suggestions based on that.

[1266] "A means of scanning products in a physical store and combining it with the information on ingredients in the home refrigerator to suggest the optimal menu" is a system that uses a device such as a smartphone to scan products in the store, and then links that information with the information on ingredients in the home refrigerator to suggest the optimal menu.

[1267] "Means for analyzing preferences and nutritional needs and proposing balanced meals" refers to technology that analyzes the preferences and nutritional status of individuals and families, and then proposes nutritionally balanced meals based on that analysis.

[1268] The present invention is a meal recommendation system that integrates a user's emotional state with information about ingredients in the refrigerator and provides shopping support in physical stores. This system is mainly composed of a server and a user terminal. Specific embodiments are as follows.

[1269] System Configuration

[1270] 1. Server

[1271] The server is a back-end system that includes an image recognition means, a database, a recipe generation means, a plating support means, a shopping list generation means, a preference analysis means, and an emotion engine.

[1272] 2. User Device

[1273] The user terminal is a device such as a smartphone or tablet that can be used to take photos of the food in the refrigerator and send them to the server. It also has a function for scanning products in physical stores.

[1274] Program processing

[1275] 1. Image Recognition

[1276] Users take photos of the inside of their refrigerator with their smartphones and upload them to the system. The server analyzes the received images and identifies the ingredients in the refrigerator in real time. It uses an image recognition algorithm (e.g., OpenCV) to extract the name and quantity of each ingredient and stores them in a database.

[1277] 2. User Profile and Preference Analysis

[1278] Users input their preferences, health status, and seasonal information through their terminals. This information is sent to the server, and the user profile is updated. This profile is then analyzed using the preference analysis means.

[1279] 3. Emotion Engine

[1280] The emotion engine detects the user's emotional state through voice input and facial expression recognition. For example, if a user says "I'm feeling a little stressed," this information is analyzed.

[1281] 4. Recipe Generation

[1282] The server generates optimal menus and recipes based on the user profile and the identified ingredient information. Emotional information generated by the emotion engine is also taken into account. The generated menus and recipes include both those that use only available ingredients and those that supplement ingredients that are lacking. The server also suggests ways to present the food after cooking and how to cut the ingredients.

[1283] 5. In-store scanning

[1284] When users shop at a physical store, they scan products with their smartphones. The server analyzes the images and lists any missing ingredients that need to be purchased. It also combines the scanned products with information about ingredients at home to suggest optimal meal plans.

[1285] 6. Shopping List Generation

[1286] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[1287] Specific examples

[1288] For example, if a user wants to use tomatoes and chicken from their refrigerator, they can take a picture of the refrigerator with their smartphone and upload it. They can also say, "I'm feeling a little stressed." Based on this information, the server will suggest low-carb recipes using tomatoes that will help reduce stress. When in a physical store, they can scan the products in the store with the app and easily see what ingredients they need to buy.

[1289] Prompt Sentence Examples

[1290] 1. Prompt (Image Recognition): "The user has uploaded a photo of the inside of their refrigerator. Please recognize the ingredients in this image and store them in a database."

[1291] 2. Prompt (Emotion Recognition): "Please analyze the emotion from the user's voice input. In this case, the input is 'I feel a little stressed.'"

[1292] 3. Prompt (Recipe Generation): "Based on the stored ingredient data and the user's sentiment data, please suggest colorful and nutritious dishes."

[1293] 4. Prompt (in-store): "Scan the products in the store and combine them with the ingredients in your refrigerator at home to suggest the perfect meal plan."

[1294] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1295] Step 1:

[1296] The user takes a photo of the inside of the refrigerator with the device and uploads it to the system. The input is the image of the inside of the refrigerator, and the output is image data sent to the server. At this time, the device uses its camera function to obtain the image data and sends it to the server via the network.

[1297] Step 2:

[1298] The server analyzes the received images and identifies the ingredients in the refrigerator in real time. The input is the uploaded image data, and the output is the identified ingredient information. The server uses an image recognition algorithm (e.g., OpenCV) to analyze the ingredients in the image, extract the names and quantities, and store them in a database.

[1299] Step 3:

[1300] The user inputs their preferences, health status, and seasonal information through the device. The input is the user's personal information, and the output is profile data sent to the server. The device then fills in a form with the data entered by the user and sends it to the server.

[1301] Step 4:

[1302] The server updates the user profile based on the received preference and health information. The input is the user's profile data, and the output is the updated user profile. The server accesses the database and updates the existing profile with the new information.

[1303] Step 5:

[1304] The emotion engine detects the user's emotional state through voice input and facial expression recognition. The input is the user's voice and image data, and the output is the detected emotional information. The server uses voice recognition technology (e.g., Google Cloud Speech-to-Text) and facial expression recognition technology to analyze the user's current emotion and store the information in a database.

[1305] Step 6:

[1306] The server generates optimal menus and recipes based on the updated user profile, identified ingredient information, and emotion information. The inputs are profile data, ingredient data, and emotion data, and the output is the generated menus and recipes. The server integrates this data and uses a recipe generation algorithm to suggest optimal menus.

[1307] Step 7:

[1308] The generated menu and recipes also include information on presentation and cutting methods. The input is the generated menu, and the output is detailed cooking instructions. The server generates detailed information on presentation and cutting methods for the recipe to provide dishes that are easy for users to understand, and sends it to the user's terminal.

[1309] Step 8:

[1310] When a user scans an item in a physical store, the terminal takes a photo of the item and sends it to the server. The input is the product image in the store, and the output is image data sent to the server. The terminal scans the item in the store and sends the scanned image to the server.

[1311] Step 9:

[1312] The server analyzes the received product images, identifies what has been scanned, lists the missing ingredients, and notifies the user. The input is the product image and information about ingredients in the refrigerator, and the output is a list of missing ingredients. The server uses image recognition technology to analyze the scanned product information, compares it with data from the home refrigerator to identify missing ingredients, and automatically generates the list and sends it to the user's device.

[1313] Step 10:

[1314] When missing ingredients are identified, the server automatically generates a shopping list and sends it to the user. The input is information about the missing ingredients, and the output is a shopping list. The server generates a shopping list including a purchase link based on the list of missing ingredients and sends it to the user's terminal.

[1315] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1316] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1317] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1318] [Fourth embodiment]

[1319] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1320] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1321] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1322] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1323] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1324] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1325] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1326] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1327] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1328] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1329] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1330] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1331] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1332] The present invention is a system for managing ingredients stored in a user's refrigerator and proposing menus and recipes based on the ingredients, and is comprised of a server and a user terminal. Specific embodiments of the present invention will be described below.

[1333] System Configuration

[1334] The server is a back-end system that includes image recognition, a database, recipe generation, plating assistance, shopping list generation, and preference analysis. The server processes data sent from the user's device and generates the necessary information.

[1335] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server, and corresponds to a smartphone, tablet, etc. An application is installed on the terminal, and the user interacts with the system through this application.

[1336] Program processing overview

[1337] First, the user takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. At this time, it uses an image recognition algorithm to extract the name and quantity of each ingredient and stores them in a database.

[1338] Next, the user inputs their preferences, health status, and dietary information via their device. This information is sent to the server, which updates the user profile. The server then generates optimal menus and recipes based on the user profile and the identified ingredients.

[1339] The generated menus and recipes include both those that use only available ingredients and those that can be realized by supplementing ingredients that are in short supply. In addition, suggestions are also made on how to present the food after cooking and how to cut the ingredients, and this information is sent to the user's device.

[1340] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[1341] Finally, the server analyzes the preferences and nutritional needs of each family member and suggests balanced meals that everyone can enjoy. This information is also sent to the device, allowing the user to easily implement the suggested menu. The server also provides information on drink pairings, dining out menus, and restaurant recommendations.

[1342] Specific examples

[1343] For example, a user takes a photo of the inside of their refrigerator with their smartphone and uploads it to a server through an application. The server identifies the presence of tomatoes, chicken, and spinach in the image. The user then enters their personal information, such as "I want a low-carb diet" and "I like tomato dishes."

[1344] The server uses this information to suggest low-carb recipes, such as tomato chicken stew, and provides presentation and cutting instructions. If you need more cream than the spinach you need, the server adds it to your shopping list and notifies you.

[1345] Users can follow suggested recipes and beautifully arrange the food on a plate using the provided plating methods. They can also get recommendations for white wine pairings and information on nearby healthy restaurants.

[1346] In this way, the system of the present invention supports the user in planning their meals and provides a healthy and varied diet.

[1347] The processing flow will be explained below.

[1348] Step 1:

[1349] The user takes a photo of the food items in the refrigerator using their smartphone (terminal).

[1350] Specific operation: The user launches the camera app on their smartphone and takes a picture of the ingredients in the refrigerator.

[1351] Step 2:

[1352] The terminal uploads the captured photos to the server.

[1353] Specific operation: The device sends the captured photo data to the specified API endpoint.

[1354] Step 3:

[1355] The server analyzes the received image data and identifies the ingredients.

[1356] Specific operation: The server uses an image recognition algorithm (e.g., Convolutional Neural Network) to identify the type and quantity of ingredients.

[1357] Step 4:

[1358] The server stores the identified ingredient information in a database.

[1359] Specific operation: The extracted list of ingredients and their quantities are associated with the user's profile and recorded in the database.

[1360] Step 5:

[1361] Users input their preferences, health status, and food intake information via the terminal.

[1362] Specific operation: The user enters various information into a form within the application and presses the submit button.

[1363] Step 6:

[1364] The terminal transmits the input user information to the server.

[1365] Specific operation: The device sends the entered preference and health status data to the server via API.

[1366] Step 7:

[1367] The server adds or updates the received user information to the profile.

[1368] Specific operations: The server merges the existing user profile with the newly received information and updates the database.

[1369] Step 8:

[1370] The server integrates the information about ingredients in the refrigerator with the updated user profile to generate menus and recipes.

[1371] What it does: Uses an AI recommendation system to suggest optimal menus and recipes based on user profile and available ingredients.

[1372] Step 9:

[1373] The server sends the generated menu and recipes to the terminal.

[1374] Specific operation: The proposed recipe and its cooking instructions are sent to the device in JSON format via API.

[1375] Step 10:

[1376] The server generates images of how to arrange and cut the food based on the proposed menu.

[1377] Specific operations: Generate presentation guides and visual materials after cooking and prepare them as image data.

[1378] Step 11:

[1379] The server sends the presentation image and guide information to the terminal.

[1380] Specific operation: The generated presentation guide and images are sent to the device via API.

[1381] Step 12:

[1382] The server lists the ingredients that are missing and generates a shopping list.

[1383] Specific behavior: Compare the inventory in the refrigerator with the suggested recipe and make a list of missing ingredients.

[1384] Step 13:

[1385] The server transmits the shopping list to the terminal.

[1386] Specific operation: The list of ingredients that are in short supply and their purchase links are sent to the device via API.

[1387] Step 14:

[1388] The server analyzes the preferences and nutritional needs of each family member and generates a balanced menu.

[1389] Specific operation: The AI ​​references each user profile and generates a menu that can be enjoyed by all.

[1390] Step 15:

[1391] The server sends the proposed balanced menu to the terminal.

[1392] Specific operation: The generated menu and its procedures are sent to the device via API.

[1393] Step 16:

[1394] The server will suggest drink pairings to go with the menu, dining out menus, and recommended restaurants.

[1395] What it does: Recommends drinks and dining options based on a suggested menu and creates a list.

[1396] Step 17:

[1397] The server transmits this additional information to the terminal and notifies the user.

[1398] Specific operation: Recommendation information is sent to the device via API and notified to the user.

[1399] Example 1

[1400] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1401] In today's busy lifestyles, efficiently managing ingredients and planning meals is a difficult task. Properly managing ingredients in the refrigerator and providing menus and recipes that match the user's preferences and health status are important for achieving a healthy and diverse diet. However, conventional systems do not adequately address these needs, forcing users to spend time managing ingredients and searching for recipes. Furthermore, it is difficult to provide suggestions that balance the preferences and nutritional needs of each family member.

[1402] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1403] In this invention, the server includes an image recognition unit for identifying ingredients in the user's refrigerator, a unit for integrating the identified ingredient information with a user profile, a unit using a generative AI model to generate menus and recipes based on the ingredient information and the user profile, a unit using an image generation algorithm to generate information on presentation and cutting methods related to the menus and recipes, a unit for sending the generated information to a user terminal, a unit for listing missing ingredients and automatically generating a shopping list, a unit for analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, and a unit for suggesting drink pairings, restaurant menus, and recommended restaurants. This allows for efficient management of ingredients in the refrigerator and automatically generating optimal menus and recipes tailored to the user's preferences and health status. It also provides balanced meal suggestions and automatic purchasing assistance for missing ingredients.

[1404] "Image recognition means" refers to means that includes algorithms and techniques for identifying ingredients in the refrigerator.

[1405] A "user profile" is a collection of personal information related to a user, such as the user's health status, preferences, and seasonal information.

[1406] A "generative AI model" is an artificial intelligence model that automatically generates menus and recipes based on ingredient information and user profiles.

[1407] An "image generation algorithm" is an algorithm for generating information on presentation and cutting methods related to menus and recipes in the form of realistic images.

[1408] A "user terminal" is a device that a user uses to interact with the system, such as a smartphone or tablet.

[1409] The "shopping list generating means" is a means for listing ingredients that are in short supply and automatically generating a shopping list.

[1410] The "preference analysis tool" is a tool for analyzing the preferences and nutritional needs of each family member and proposing balanced meals.

[1411] The "pairing suggestion means" is a means for suggesting drink pairings, dining out menus, and information on recommended restaurants.

[1412] MODE FOR CARRYING OUT THE INVENTION

[1413] The present invention is a system that manages ingredients stored in a user's refrigerator and suggests menus and recipes based on the ingredients. The system is composed of a server and a user terminal. Specific embodiments of the present invention are described below.

[1414] The server is a backend system that includes image recognition means, a database, recipe generation means, plating assistance means, shopping list generation means, and preference analysis means. The server processes data sent from the user's device and generates the necessary information. For server hardware, a general-purpose server machine (e.g., a server equipped with a high-performance CPU, sufficient memory, and storage) is used. For software, image recognition algorithms such as "YOLO" or "Google Cloud Vision API" are used, databases such as "MySQL," "Firebase," or "AWS DynamoDB" are used, and generative AI models such as "GPT-3" are used.

[1415] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server; it can be a smartphone or tablet. A dedicated application is installed on the terminal, and the user interacts with the system through this application. This application has functions for taking pictures, entering data, and receiving notifications. The terminal hardware is a smartphone with a camera (e.g., Android or iPhone). The software is a dedicated mobile application. This app provides a user interface that allows the user to upload images, enter profile information, and receive notifications.

[1416] The user first takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. To do this, the user opens the refrigerator door and takes a photo of the entire interior with the smartphone camera. Next, the user taps the "Upload" button on the application to send the image. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. For the analysis, it uses the image recognition algorithm "YOLO" and "Google Cloud Vision API." The server then stores the names and quantities of the identified ingredients in a database. This database uses "MySQL," "Firebase," or "AWS DynamoDB."

[1417] Next, the user enters their preferences, health status, and dietary information via their device. For example, if the user prefers a "low-carb" or "low-fat" diet, they enter this information into a form within the app and tap the "Submit" button. The server receives this information and updates the user profile. The server then generates optimal menus and recipes based on the updated user profile and the identified ingredient information. In this process, a generative AI model such as "GPT-3" is used to generate recipes that take into account the user's preferences and available ingredients.

[1418] In addition to the recipe, the server also suggests how to arrange the food after cooking and how to cut the ingredients. To do this, it uses an "image generation AI model" (e.g., DALL-E) to generate a sample image of the presentation and sends it to the user's device. If any ingredients are missing, the server lists them and automatically generates a shopping list. This shopping list also includes a link to purchase the food online and notifies the user.

[1419] The server also analyzes the preferences and nutritional needs of each family member and suggests balanced meals. This analysis could potentially use the Human Nutrition API. It also provides information on drink pairings, dining out menus, and recommended restaurants. Drink pairing information can be obtained from the Wine.com API, while restaurant information can be obtained from the Google Places API or Yelp API.

[1420] Specific examples

[1421] For example, a user can take a photo of the contents of their refrigerator with their smartphone and upload it to a server through an application. The server can identify the presence of tomatoes, chicken, and spinach in the image. The user then enters information into the app, such as "I'm looking for a low-carb diet" and "I like tomato dishes." Based on this information, the server can suggest a low-carb recipe for tomato chicken stew. If the user has the spinach needed for this recipe in their refrigerator but is short on cream, the server will add the cream to their shopping list and notify them. The user can then cook the dish according to the suggested recipe and arrange it on a plate using the suggested plating method. The server can also suggest white wine pairings and provide information about nearby healthy restaurants.

[1422] Prompt example

[1423] "You take a photo of the contents of your refrigerator and upload it to the system. It identifies that you have tomatoes, chicken, and spinach, and requests a low-carb meal and a tomato dish."

[1424] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1425] Program processing flow

[1426] Step 1:

[1427] Users take photos of the inside of the refrigerator with their smartphone (device) and upload them to the server through the application. Specifically, the user opens the refrigerator door and takes a photo of the entire interior with the smartphone camera. Next, they tap the "upload" button in the application to send the image.

[1428] Input: Image of the inside of the refrigerator

[1429] Output: Image data sent to the server

[1430] Step 2:

[1431] The server analyzes the received images and identifies the ingredients in the refrigerator in real time. It uses an image recognition algorithm (e.g., YOLO or Google Cloud Vision API) to extract the names and quantities of ingredients and store them in a database. Specifically, after receiving the images, the server runs the image recognition software and stores the identified ingredient data in MySQL, Firebase, or AWS DynamoDB.

[1432] Input: Image data uploaded by the user

[1433] Output: Database update information including ingredient names and quantities

[1434] Step 3:

[1435] Users input their preferences, health status, and dietary information via their device. Specifically, the user enters information such as "low-carb" or "low-fat" into the application's input form and taps the "Submit" button. The server receives this information and updates the user profile.

[1436] Input: User preferences and health data

[1437] Output: Updated user profile

[1438] Step 4:

[1439] The server generates optimal menus and recipes based on the updated user profile and identified ingredient information. Specifically, the server uses a generative AI model (e.g., GPT-3) to generate recipes that take into account the user's preferences and available ingredients.

[1440] Input: Updated user profile and ingredient information

[1441] Output: Generated menu and recipes

[1442] Step 5:

[1443] The server then suggests post-cooking plating and cutting methods related to the generated menu and recipe. Specifically, the server uses an image generation AI model (e.g., DALL-E) to generate sample plating images and sends this information to the user's device.

[1444] Input: Generated menu and recipe

[1445] Output: Suggestions for presentation and cutting of ingredients

[1446] Step 6:

[1447] The server lists the ingredients that are in short supply and automatically generates a shopping list. Specifically, the server checks for ingredients that are in short supply, creates a shopping list using the Amazon Web Services Lambda function, and notifies the user with a link to purchase the ingredients.

[1448] Input: Ingredient information from the database and the generated recipe

[1449] Output: List of ingredients that are in short supply and a shopping list

[1450] Step 7:

[1451] The server analyzes the preferences and nutritional needs of each family member and suggests balanced meals. Specifically, the server uses the "Human Nutrition API" to calculate nutritional information and compare it with the preference information of each family member. In addition, the server also provides drink pairings and recommended restaurant information.

[1452] Input: Data on the preferences and health status of all family members

[1453] Output: Balanced meal suggestions, drink pairings and restaurant recommendations

[1454] (Application example 1)

[1455] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1456] Conventional food management systems were able to identify ingredients in the refrigerator and suggest recipes based on the user's preferences and health information. However, they were unable to automatically process the entire process, from purchasing missing ingredients to delivery. Furthermore, they lacked optimal recipe suggestions using generative AI models, making it impossible to achieve the high level of personalization desired by users. The present invention aims to solve these issues and provide users with a consistent, personalized meal suggestion and delivery service.

[1457] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1458] In this invention, the server includes an image recognition unit for identifying ingredients in the user's refrigerator, a unit for integrating the identified ingredient information with a user profile, a unit for generating menus and recipes based on the ingredient information and the user profile, a unit for generating information on plating and cutting methods related to the menus and recipes, a unit for sending the generated information to a user terminal, a unit for listing missing ingredients and automatically generating a shopping list, a unit for analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, a unit for suggesting drink pairings and restaurant menus, a unit for automatically selecting missing ingredients based on the identified ingredient information and delivering them together with cooking, and a unit for generating optimal dishes and recipes using a generative AI model. This allows users to manage ingredients in their refrigerator in real time, receive automatic delivery of missing ingredients, and enjoy highly personalized menus and dishes.

[1459] "Image recognition means for identifying ingredients in a user's refrigerator" refers to technology for analyzing images of the inside of a refrigerator taken by a user and identifying ingredients.

[1460] "Means for integrating the identified ingredient information with a user profile" refers to a method for integrating ingredient information identified in a refrigerator with user preferences, health information, etc.

[1461] "Means for generating menus and recipes based on the ingredient information and user profile" refers to technology for generating optimal menus and recipes based on identified ingredient information and user profile information.

[1462] The term "means for generating information on presentation and cutting methods related to the menu and recipe" refers to a method for generating information on presentation and cutting methods for the proposed menu and recipe.

[1463] "Means for transmitting the generated information to the user terminal" refers to a communication method for transmitting information such as the generated menu, recipes, and presentation methods to the user terminal.

[1464] "Means for listing missing ingredients and automatically generating a shopping list" refers to a method for listing missing ingredients required for a proposed recipe and automatically generating a shopping list.

[1465] "Means for analyzing the preferences and nutritional needs of all family members and suggesting balanced meals" refers to a method for analyzing the preferences and nutritional needs of all family members and suggesting balanced meals based on that analysis.

[1466] "Means for suggesting drink pairings and restaurant menus" refers to methods for recommending drinks that go well with the proposed dishes and restaurant menus.

[1467] "Means for automatically picking up missing ingredients based on the identified ingredient information and delivering them together with cooking" refers to a method for automatically picking up missing ingredients based on identified ingredient information and delivering them together with cooking.

[1468] "Means for generating optimal dishes and recipes using a generative AI model" refers to technology that uses a generative AI model to generate optimal dishes and recipes based on the user's preferences and information about ingredients in the refrigerator.

[1469] The present invention relates to a system that manages ingredients in a user's refrigerator and proposes menus and recipes based on the user's preferences and health information. This allows the user to receive personalized meal suggestions, including automatic delivery of ingredients that the user is running low on. Specific embodiments of the present invention are described below.

[1470] System Configuration

[1471] This system mainly consists of a server and a terminal.

[1472] 1. Server:

[1473] The server is a backend system that includes an image recognition means, a database, a recipe generation means, a plating support means, a shopping list generation means, a preference analysis means, and a generative AI model.

[1474] The server has the function of processing data sent from the user's terminal and generating the necessary information.

[1475] 2. Terminal:

[1476] The terminal is a device that allows the user to take a photo of the inside of the refrigerator and send it to the server. This can be a smartphone or tablet.

[1477] Applications are installed on the terminals, and users interact with the system through these applications.

[1478] Program processing overview

[1479] 1. Image Recognition Methods:

[1480] First, the user takes a photo of the inside of the refrigerator using their smartphone (terminal) and uploads it to the system.

[1481] The server analyzes the received images and identifies the ingredients in the refrigerator in real time, using an image recognition algorithm (e.g., TensorFlow) to extract the name and quantity of each ingredient and store them in a database.

[1482] 2. Data integration and recipe generation:

[1483] Users input their preferences, health status, and dietary information through their device, which is then sent to the server and the user profile is updated.

[1484] The server generates optimal menus and recipes based on the user profile and identified ingredient information. It generates dishes and recipes using a generative AI model (e.g., a generative AI model).

[1485] 3. Presentation and Delivery:

[1486] The generated menus and recipes include both those that use only available ingredients and those that can be achieved by supplementing missing ingredients.

[1487] Suggestions are also given on how to arrange the food after cooking and how to cut the ingredients, and this information is sent to the user's terminal.

[1488] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[1489] The server also provides a means to automatically pick up missing ingredients and deliver them along with cooking.

[1490] 4. Meeting the tastes and nutritional needs of the whole family:

[1491] It analyzes the preferences and nutritional needs of each family member and suggests balanced meals that everyone can enjoy. This information is also sent to the device, allowing the user to easily put the suggested menu into practice.

[1492] The server also provides information on drink pairings, dining out menus, and recommended restaurants.

[1493] Specific examples

[1494] For example, a user can take a photo of the contents of their refrigerator with their smartphone and upload it to a server through an application. The server can identify the presence of tomatoes, chicken, and spinach in the image. The user then inputs their personal information, such as "I'm looking for a low-carb diet" and "I like tomato dishes." The server then uses this information to suggest low-carb recipes, such as tomato chicken stew. It also provides an image of how the food will look after cooking and how to cut it. If the user needs more cream than the spinach they need, the server adds it to their shopping list and notifies them. The user can then prepare the dish according to the suggested recipe and beautifully arrange it on a plate using the provided plating method. They can also receive recommendations for white wines to pair with their drinks and information about nearby healthy restaurants.

[1495] Prompt Sentence Examples

[1496] The following prompt sentence is used as input to the generative AI model:

[1497] I have tomatoes and chicken in my fridge. Analyze images of the fridge and suggest low-calorie meals using these ingredients. If the suggested meal contains missing ingredients, add them to the shopping list and offer them as a delivery option.

[1498] The above is an embodiment of the present invention, which allows users to efficiently manage ingredients in their refrigerators and receive personalized meal suggestions and delivery services.

[1499] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1500] Step 1:

[1501] The user takes a photo of the inside of the refrigerator using their smartphone (device). The image is saved on the device and becomes the input data for the system.

[1502] Step 2:

[1503] The user uploads a photo of the inside of the refrigerator to the server via the terminal. The terminal sends the image data to the server via the Internet. In this case, the image file becomes the input data.

[1504] Step 3:

[1505] The server analyzes the received images and identifies the ingredients in the refrigerator in real time. Specifically, the server uses an image recognition algorithm (e.g., TensorFlow) to extract the name and quantity of each ingredient and store the identified ingredient information in a database. In this process, image data is the input and identified ingredient information is the output.

[1506] Step 4:

[1507] Users input their preferences, health status, and dietary information through their device. This information is sent from the device to the server, where the user profile is updated. Here, preferences and health information are input data, and the updated user profile is output.

[1508] Step 5:

[1509] The server generates optimal menus and recipes based on the user profile and identified ingredient information. The server uses a generative AI model (e.g., generative AI model) to generate dishes and recipes based on the prompt. In this process, the user profile and ingredient information are input data, and the generated menus and recipes are output.

[1510] Step 6:

[1511] The server generates information on how to arrange and cut ingredients related to the menu and recipe, allowing the user to receive specific cooking and presentation advice. In this process, the menu and recipe are input data, and information on how to arrange and cut ingredients is output.

[1512] Step 7:

[1513] The server sends the generated information to the user's terminal, where the user can view the proposed menu, recipes, presentation methods, and other information. In this process, all the information generated up to that point is input data, and the data sent to the user's terminal is the output.

[1514] Step 8:

[1515] The server lists the missing ingredients and automatically generates a shopping list. Furthermore, this list also includes a purchase link so that the user can easily purchase the missing ingredients. In this process, the required ingredient information from the menu and recipes is the input data, and the shopping list is the output.

[1516] Step 9:

[1517] The server automatically picks up missing ingredients based on the identified ingredient information, cooks them, and delivers them. Specifically, an order is issued to a partner delivery service. In this process, the missing ingredient information is the input data, and a delivery order is the output.

[1518] Step 10:

[1519] The server analyzes the preferences and nutritional needs of each family member and suggests balanced meals. It also suggests drink pairings and restaurant menus, and users can receive this information on their devices. In this process, the input data is the profiles of each family member, and the output is balanced meals and pairing information.

[1520] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1521] This system manages the ingredients in a user's refrigerator and suggests menus and recipes based on that information. In particular, by combining it with an emotion engine, this system makes it possible to suggest meals based on the user's emotions. It is composed of a server and a user terminal.

[1522] System Configuration

[1523] The server is a back-end system that includes image recognition, a database, recipe generation, plating assistance, shopping list generation, preference analysis, and an emotion engine. The server processes data sent from the user's device and generates the necessary information.

[1524] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server, and corresponds to a smartphone, tablet, etc. An application is installed on the terminal, and the user interacts with the system through this application.

[1525] Program processing overview

[1526] First, the user takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. At this time, it uses an image recognition algorithm to extract the name and quantity of each ingredient and stores them in a database.

[1527] Next, the user inputs their preferences, health status, and dietary information via their device. This information is sent to the server, which updates the user profile. The server then generates optimal menus and recipes based on the user profile and the identified ingredients.

[1528] The generated menus and recipes include both those that use only available ingredients and those that can be realized by supplementing ingredients that are in short supply. In addition, suggestions are also made on how to present the food after cooking and how to cut the ingredients, and this information is sent to the user's device.

[1529] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[1530] The server analyzes the preferences and nutritional needs of each family member to suggest balanced meals that everyone can enjoy. This information is also sent to the device, allowing users to easily implement the suggested menu. The server also provides information on drink pairings, dining out menus, and recommended restaurants.

[1531] Emotion engine integration

[1532] This system uses an emotion engine to recognize the user's emotions in real time. The emotion engine can detect the user's emotional state through voice input and facial expression recognition. This emotional information is directly reflected in the menu and recipe suggestions generated by the server.

[1533] For example, if a user is feeling down, the emotion engine can detect this information and suggest colorful, nutritious dishes to lift their spirits. On the other hand, if the user is tired, it can suggest recipes that are easy to prepare or that use ingredients that have a relaxing effect. In this way, personalized meal suggestions based on the emotion engine are possible.

[1534] Specific examples

[1535] For example, a user can take a photo of the inside of their refrigerator with their smartphone and upload it to a server through an application, which can then identify the presence of tomatoes, chicken, and spinach in the image.

[1536] Next, the user inputs their preferences, such as "I want a low-carb diet" and "I like tomato dishes." The emotion engine also detects emotional information from the user's voice input, such as "I'm feeling a little stressed."

[1537] The server uses this information to suggest low-carb recipes, such as tomato chicken stew, that contain ingredients that have stress-reducing properties. The suggestions also include presentation and cutting instructions. If cream is the missing ingredient, the server adds it to the shopping list and notifies the user.

[1538] Users can cook according to the suggested recipes and beautifully arrange the food on a plate using the provided plating methods. They can also get information on relaxing herbal teas to pair with their drinks, as well as healthy restaurants nearby.

[1539] In this way, the system of the present invention integrates the user's emotions with the management of ingredients in the refrigerator, enabling the provision of personalized, healthy, and diverse meals.

[1540] The processing flow will be explained below.

[1541] Step 1:

[1542] The user takes a photo of the food items in the refrigerator using their smartphone (terminal).

[1543] Specific operation: The user launches the camera app on their smartphone and takes a picture of the ingredients in the refrigerator.

[1544] Step 2:

[1545] The terminal uploads the captured photos to the server.

[1546] Specific operation: The device sends the captured photo data to the specified API endpoint.

[1547] Step 3:

[1548] The server analyzes the received image data and identifies the ingredients.

[1549] Specific operation: The server uses an image recognition algorithm (e.g., Convolutional Neural Network) to identify the type and quantity of ingredients.

[1550] Step 4:

[1551] The server stores the identified ingredient information in a database.

[1552] Specific operation: The extracted list of ingredients and their quantities are associated with the user's profile and recorded in the database.

[1553] Step 5:

[1554] Users input their preferences, health status, and food intake information via the terminal.

[1555] Specific operation: The user enters various information into a form within the application and presses the submit button.

[1556] Step 6:

[1557] The terminal transmits the input user information to the server.

[1558] Specific operation: The device sends the entered preference and health status data to the server via API.

[1559] Step 7:

[1560] The server adds or updates the received user information to the profile.

[1561] Specific operations: The server merges the existing user profile with the newly received information and updates the database.

[1562] Step 8:

[1563] The device collects voice input and facial expression images to detect the user's emotions.

[1564] Specific operation: The terminal records the user's voice or saves a photograph of the user's face.

[1565] Step 9:

[1566] The terminal uses an emotion engine to analyze the user's emotions.

[1567] Specific behavior: Estimates the user's emotional state using voice and facial expression analysis algorithms.

[1568] Step 10:

[1569] The terminal transmits the analysis results to the server.

[1570] Specific operation: The estimated emotion information is sent to the server via API.

[1571] Step 11:

[1572] The server integrates information about ingredients in the refrigerator with the updated user profile and emotional information to generate menus and recipes.

[1573] What it does: Uses an AI recommendation system to suggest optimal menus and recipes based on user profile, available ingredients, and emotional information.

[1574] Step 12:

[1575] The server sends the generated menu and recipes to the terminal.

[1576] Specific operation: The proposed recipe and its cooking instructions are sent to the device in JSON format via API.

[1577] Step 13:

[1578] The server generates images of how to arrange and cut the food based on the proposed menu.

[1579] Specific operations: Generate presentation guides and visual materials after cooking and prepare them as image data.

[1580] Step 14:

[1581] The server sends the presentation image and guide information to the terminal.

[1582] Specific operation: The generated presentation guide and images are sent to the device via API.

[1583] Step 15:

[1584] The server lists the ingredients that are missing and generates a shopping list.

[1585] Specific behavior: Compare the inventory in the refrigerator with the suggested recipe and make a list of missing ingredients.

[1586] Step 16:

[1587] The server transmits the shopping list to the terminal.

[1588] Specific operation: The list of ingredients that are in short supply and their purchase links are sent to the device via API.

[1589] Step 17:

[1590] The server analyzes the preferences and nutritional needs of each family member and generates a balanced menu.

[1591] Specific operation: The AI ​​references each user profile and generates a menu that can be enjoyed by all.

[1592] Step 18:

[1593] The server sends the proposed balanced menu to the terminal.

[1594] Specific operation: The generated menu and its procedures are sent to the device via API.

[1595] Step 19:

[1596] The server will suggest drink pairings to go with the menu, dining out menus, and recommended restaurants.

[1597] What it does: Recommends drinks and dining options based on a suggested menu and creates a list.

[1598] Step 20:

[1599] The server transmits this additional information to the terminal and notifies the user.

[1600] Specific operation: Recommendation information is sent to the device via API and notified to the user.

[1601] Example 2

[1602] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1603] In modern life, there is a demand for efficient management of ingredients in the refrigerator and for optimal menus and recipes to be proposed based on the user's preferences and health status. However, conventional systems have limitations in ingredient management and menu proposals, and are unable to realize personalized meal proposals that reflect the user's emotional state. Furthermore, there are insufficient means for grasping ingredient inventory status and efficiently replenishing ingredients that are in short supply. This leads to problems such as unbalanced meals and food waste. Therefore, there is a need for a system that can solve these issues and propose more personalized and balanced meals.

[1604] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1605] In this invention, the server includes an image recognition means for identifying ingredients in the user's refrigerator, a means for integrating the identified ingredient information with a user profile, a means for generating menus and recipes based on the ingredient information and the user profile, a means for generating information on presentation and cutting methods related to the menus and recipes, a means for transmitting the generated information to a user terminal, a means for listing ingredients that are in short supply and automatically generating a shopping list, a means for analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, a means for suggesting drink pairings and restaurant menus, and a sentiment analysis means for analyzing the user's emotional state and generating menus and recipes based on the sentiment information. This enables the system to suggest optimal menus and recipes based on the user's emotions and health condition, and to efficiently manage ingredients accordingly.

[1606] "Image recognition means" refers to the technology and algorithms used to analyze images of the inside of the refrigerator taken by the user and identify the ingredients present.

[1607] "User profile" refers to an individual data set that includes a user's preferences, health status, dietary intake information, seasonal information, etc.

[1608] "Means for generating menus and recipes" refers to algorithms and processes for generating appropriate menus and specific recipes based on the user profile and information about ingredients in the refrigerator.

[1609] The term "means for generating information on presentation and cutting methods" refers to a technology that generates specific instructions on how to present and cut ingredients after cooking in relation to the generated menu and recipe.

[1610] "Means for sending information to a user's terminal" refers to the technology for sending various information generated on the server side (menus, recipes, presentation methods, etc.) to a user's terminal such as a smartphone or tablet via the Internet.

[1611] "Shopping list generation means" refers to a technology and process that automatically lists ingredients that are in short supply based on information about ingredients in the refrigerator and the generated recipe, and provides the list to the user.

[1612] "Means for analyzing preferences and nutritional needs" refers to algorithms and technologies for collecting and analyzing the preferences and nutritional needs of each family member and suggesting balanced meals.

[1613] "Means for suggesting drink pairings and restaurant menus" refers to the algorithms and technologies for suggesting drinks that match the generated menu and restaurant menus.

[1614] "Emotion analysis means" refers to the technology and algorithms used to analyze a user's emotional state and generate menus and recipes based on that information.

[1615] This system manages the ingredients in a user's refrigerator and suggests menus and recipes based on that information. By combining it with an emotion engine, it is possible to suggest meals that correspond to the user's emotions. The system is composed of a server and a user terminal.

[1616] System Configuration

[1617] The server is a back-end system that includes an image recognition unit, a database, a recipe generation unit, a plating support unit, a shopping list generation unit, a preference analysis unit, and an emotion engine. The server processes data sent from the user's device and generates the necessary information.

[1618] The terminal is a device that allows the user to take pictures of the inside of the refrigerator and send them to the server, and corresponds to a smartphone, tablet, etc. An application is installed on the terminal, and the user interacts with the system through this application.

[1619] Program processing overview

[1620] First, the user takes a photo of the inside of the refrigerator with their smartphone (device) and uploads it to the system. The server analyzes the received image and identifies the ingredients in the refrigerator in real time. In doing so, it uses an image recognition algorithm to extract the name and quantity of each ingredient and stores them in a database. Specifically, it uses libraries such as TensorFlow and OpenCV.

[1621] Next, the user inputs their preferences, health status, and dietary information via their device. This information is sent to the server, which updates the user profile. The server then generates optimal menus and recipes based on the user profile and the identified ingredients. The menu generation algorithm is run using Python and R libraries (scikit-learn, pandas).

[1622] The generated menus and recipes include both those that use only available ingredients and those that can be realized by supplementing ingredients that are in short supply. In addition, suggestions are also made on how to present the food after cooking and how to cut the ingredients, and this information is sent to the user's device.

[1623] If there are any missing ingredients, the server will list them and automatically generate a shopping list, which will also include a purchase link so that the user can easily purchase the missing ingredients.

[1624] The server analyzes the preferences and nutritional needs of each family member to suggest balanced meals that everyone can enjoy. This information is also sent to the device, allowing users to easily implement the suggested menu. The server also provides information on drink pairings, dining out menus, and recommended restaurants.

[1625] Emotion engine integration

[1626] This system uses an emotion engine to recognize the user's emotions in real time. The emotion engine can detect the user's emotional state through voice input and facial expression recognition. This emotional information is directly reflected in the menu and recipe suggestions generated by the server.

[1627] For example, if a user is feeling down, the emotion engine can detect this information and suggest colorful, nutritious dishes to lift their spirits. On the other hand, if the user is tired, it can suggest recipes that are easy to prepare or that use ingredients that have a relaxing effect. In this way, personalized meal suggestions based on the emotion engine are possible.

[1628] Specific examples

[1629] For example, a user takes a photo of the inside of their refrigerator with their smartphone and uploads it to a server via an application. The server identifies the presence of tomatoes, chicken, and spinach from the image. The user then inputs their preferences, such as "I would like to eat a low-carb diet" and "I like tomato dishes." The emotion engine also detects emotional information from the user's voice input, such as "I feel a little stressed."

[1630] The server uses this information to suggest low-carb recipes, such as tomato chicken stew, that contain ingredients that have stress-reducing properties. The suggestions also include presentation and cutting instructions. If the missing ingredient is cream, the server adds it to the shopping list and notifies the user.

[1631] Users can cook according to the suggested recipes and beautifully arrange the food on a plate using the provided presentation. They can also get information on relaxing herbal teas to pair with their drinks, as well as healthy restaurants nearby.

[1632] Prompt Sentence Examples

[1633] The following prompt statements can be used:

[1634] "There are tomatoes, chicken, and spinach in the refrigerator. The user wants a low-carb diet and prefers tomato dishes. There is also emotional information that the user is feeling a little stressed. Based on this information, please generate menu and recipe suggestions for the user. Also, please create a shopping list for any missing ingredients that are needed."

[1635] In this way, the system of the present invention integrates the user's emotions with the management of ingredients in the refrigerator, enabling the provision of personalized, healthy, and diverse meals.

[1636] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1637] Step 1:

[1638] Users take photos of the inside of the refrigerator with their smartphone and upload them to the server via a dedicated application.

[1639] Input: Image of the inside of the refrigerator (photo file)

[1640] Specific operation: The user takes a photo of the inside of the refrigerator using the camera function of their smartphone, then launches the application, selects the photo they took, and presses the "Send" button.

[1641] Output: Image data of the inside of the refrigerator is sent to the server.

[1642] Step 2:

[1643] The server analyzes the received image data and identifies the ingredients in the refrigerator in real time.

[1644] Input: Image data of the inside of the refrigerator

[1645] Specific operation: The server applies an image recognition algorithm (TensorFlow or OpenCV) to extract the name and quantity of each ingredient, recognize the characteristics of the ingredients, and convert them into a format that can be stored in a database.

[1646] Output: List of identified ingredients (e.g. 3 tomatoes, 500g chicken, etc.)

[1647] Step 3:

[1648] The server stores the identified ingredient information in a database.

[1649] Input: Identified ingredient list

[1650] Specific operation: The server stores the ingredient information in a database such as MySQL or PostgreSQL. For each identified ingredient, it associates the name and quantity and registers them.

[1651] Output: Updated ingredient database

[1652] Step 4:

[1653] Users enter their preferences, health status, and food intake information into the application and send it to the server.

[1654] Input: User preferences, health status, and dietary information

[1655] Specific operation: The user uses the application's input form to enter information such as "low carbohydrate diet," "prefers tomato dishes," and "high blood pressure." Then, the user presses the "Submit" button to send this information to the server.

[1656] Output: User profile information sent

[1657] Step 5:

[1658] The server uses the user profile and identified ingredient information to generate menus and recipes.

[1659] Input: User profile information, identified ingredient list

[1660] How it works: The server uses Python and R libraries (scikit-learn, pandas) to generate menus and recipes that suit the user's preferences and health status. For example, it generates optimal recipes taking into account conditions such as "low carbohydrates" and "stress reduction."

[1661] Output: Generated menu and recipe information

[1662] Step 6:

[1663] The server transmits the generated menu and recipes, as well as related information on serving and cutting methods, to the user terminal.

[1664] Input: Generated menu and recipe information

[1665] What it does: The server packages the generated information and sends it to the user's smartphone, including a recipe for "Tomato Chicken Stup" and detailed instructions on how to serve and cut the chicken.

[1666] Output: Menu and recipe information sent to the user's device

[1667] Step 7:

[1668] The server lists the ingredients that are missing and automatically generates a shopping list.

[1669] Input: Generated recipe information, current ingredient database

[1670] Specific behavior: The server compares the generated recipe with the current ingredient database to identify any missing ingredients, adds them to the list, and generates a purchase link.

[1671] Output: Shopping list

[1672] Step 8:

[1673] The server analyzes the preferences and nutritional needs of each family member and suggests balanced meals.

[1674] Input: Preferences and nutritional needs of the whole family

[1675] Specific operation: The server analyzes the collected family data and generates a menu that is nutritionally balanced and satisfies everyone.

[1676] Output: Balanced family meal plan

[1677] Step 9:

[1678] Servers will suggest drink pairings and dining out menus.

[1679] Input: Generated menu information

[1680] Specific operation: The server runs an algorithm that recommends drinks and restaurant menus that go well with the meal and generates appropriate pairing suggestions.

[1681] Output: Drink pairing and dining menu suggestions

[1682] Step 10:

[1683] The server analyzes the user's emotional state and generates menus and recipes based on the emotional information.

[1684] Input: User's emotional state (obtained from voice input, facial expression recognition, etc.)

[1685] Specific operation: The emotion engine analyzes the acquired voice and facial expression data to identify the user's emotional state. The server then regenerates menus and recipes based on parameters corresponding to the emotion.

[1686] Output: Emotion-based personalized menu and recipe information

[1687] (Application example 2)

[1688] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1689] Conventional menu suggestion systems managed the ingredients in a user's refrigerator and suggested recipes, but they did not adequately provide suggestions based on the user's emotions or instantaneous ingredient management in physical stores. As a result, their effectiveness in improving user satisfaction and reducing food waste was limited. Furthermore, they lacked personalized suggestions that took into account the user's emotions and current preferences, making it difficult to provide the meal suggestions users wanted. Furthermore, they lacked the ability to link home ingredient information when shopping in physical stores, which reduced shopping efficiency and led to wasteful purchases and duplicate ingredient purchases.

[1690] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1691] In this invention, the server includes a means for recognizing the user's emotional state and making meal suggestions based on that emotion, a means for scanning products in a physical store and proposing optimal menus in conjunction with the ingredients in the user's home refrigerator, and a means for integrating a user profile including the user's health condition, preferences, and seasonal information. This enables personalized and appropriate meal suggestions that reflect the user's emotions, health condition, and preferences in real time. Furthermore, menu suggestions can be made in conjunction with the ingredients in the user's home refrigerator even when shopping in a physical store, realizing efficient shopping and waste-free food management.

[1692] "Image recognition means" is a technology that analyzes images captured using a device such as a camera and identifies objects and text contained in the images.

[1693] A "user profile" is a data set that integrates personal information such as a user's preferences, health status, and past dietary history.

[1694] The "means for generating menus and recipes" refers to an algorithm or system that suggests optimal meal menus and cooking methods based on information about ingredients in the user's refrigerator and the user's profile.

[1695] The "means for generating information on presentation and cutting methods" is a technology for generating information to suggest beautiful presentation methods and cutting methods based on the generated recipe.

[1696] The "means for generating a shopping list" is an algorithm or system that detects ingredients that are lacking in the user's refrigerator and automatically creates a shopping list.

[1697] The "means for recognizing emotional states and making meal suggestions according to the emotions" is a system that...

Claims

1. image recognition means for identifying ingredients in the user's refrigerator; means for integrating the identified ingredient information with a user profile; A means for generating menus and recipes based on the ingredient information and a user profile; means for generating information on serving and cutting methods related to the menu and recipe; means for transmitting the generated information to a user terminal; A way to list ingredients that are missing and automatically generate a shopping list; A means of analyzing the preferences and nutritional needs of each family member and suggesting balanced meals, The system includes a means for suggesting drink pairings and dining out menus.

2. 2. The system of claim 1, wherein the image recognition means includes an algorithm for identifying ingredients in the refrigerator in real time.

3. 2. The system of claim 1, wherein the user profile includes the user's health status, preferences, and seasonal information.

Images