System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing an image of an ingredient owned by a user using an image capturing device; means for receiving the captured image and identifying the ingredient using image recognition technology; and means for providing the user with a customized recipe generated based on the identified ingredient.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Existing recipe apps and services only provide general recipes, but do not adequately address users' desire to make effective use of ingredients they have on hand, or situations where they lack cooking skills or confidence. Furthermore, they do not provide sufficient support for users who want to reduce food waste and adopt a healthier diet. For this reason, there is a demand for personalized recipes and real-time cooking support to make home cooking an efficient and enjoyable experience. [Means for solving the problem]

[0005] The present invention provides a system including a means for photographing ingredients owned by a user, a means for receiving the photographed images and identifying the ingredients using image recognition technology, and a means for providing the user with a customized recipe generated based on the identified ingredients. The system also includes a means for the user to send questions and related images in response to questions or difficulties encountered while cooking, and a means for analyzing the sent questions and images to generate appropriate solutions or advice and provide them to the user. The system further includes a means for inputting user preferences and allergy information and a means for adjusting the generation of the customized recipe taking the input preferences and allergy information into consideration, thereby providing personalized recipes and real-time cooking support, allowing the user to make the most of the ingredients they own. This makes home cooking an efficient and enjoyable experience, reduces food waste, and supports healthy eating.

[0006] "User" refers to an individual or household user who uses the system to photograph ingredients and receive customized recipes and cooking support.

[0007] "Photography device" refers to equipment with a camera function for recording images of ingredients and the cooking process.

[0008] "Image recognition technology" refers to a set of algorithms and software that analyzes transmitted image data and identifies objects contained within the image.

[0009] "Ingredients" refers to the foods and seasonings used as ingredients in cooking.

[0010] A "customized recipe" refers to cooking instructions that are individually created based on the ingredients the user has on hand and their individual requests (preferences, allergy information, etc.).

[0011] "Cooking support" refers to a service that provides appropriate advice and solutions in real time to questions and difficulties users face while cooking.

[0012] A "question" refers to a question that a user has while cooking that they verbalize and input into the system.

[0013] "Analysis" refers to the process by which the system processes the questions and images it receives and derives appropriate solutions and advice based on that data.

[0014] "Preferences" refers to the individual preferences a user has for specific ingredients or cooking methods.

[0015] "Allergy Information" refers to information indicating a user's allergic reaction to a particular food.

[0016] "Adjustment" refers to the process of changing the contents of a customized recipe based on the user's preferences and allergy information. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] System Overview

[0039] The "SmartChef" system of the present invention is a system that maximizes the use of ingredients on hand and provides personalized recipes. The system integrates image recognition technology, generative AI models, and real-time cooking support functions.

[0040] System Components

[0041] 1. Users

[0042] Use a camera (such as a smartphone camera) to take a picture of the ingredients you own.

[0043] The captured image is sent to the server via the terminal.

[0044] Take a photo of any questions or problems that arise while cooking and enter them into the device as a question.

[0045] Enter any necessary preferences or allergy information into the device.

[0046] 2. Terminal

[0047] Images taken by the user are preprocessed and sent to the server.

[0048] Display received ingredient lists and customized recipes to users.

[0049] It receives questions and related images from users while they are cooking and sends them to the server.

[0050] Display advice and guidance received from the server.

[0051] 3. Server

[0052] An image recognition algorithm is used to identify ingredients from the received image.

[0053] A customized recipe is generated based on the identified ingredient list.

[0054] Analyzes user questions and images to generate appropriate advice and solutions.

[0055] The generated ingredient list, customized recipe, and advice are sent back to the device.

[0056] The contents of the customized recipe are adjusted taking into account the user's preferences and allergy information.

[0057] Specific processing flow

[0058] Image recognition and food ingredient identification

[0059] 1. The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo with their smartphone camera.

[0060] 2. The device preprocesses the photo (adjusts resolution, converts format, etc.) and sends it to the server.

[0061] 3. The server receives the submitted image and applies image recognition algorithms to identify the ingredients in the photo, such as tomatoes, chicken, and carrots.

[0062] 4. The server generates a list of the identified ingredients and returns it to the terminal.

[0063] Customized recipe generation

[0064] 1. The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[0065] 2. The terminal sends the ingredient list and the user's preferences and allergy information to the server.

[0066] 3. The server uses the generative AI model based on the received information to generate a customized recipe, such as a "Tomato and Chicken Stew" recipe.

[0067] 4. The server returns the generated recipe (ingredients, steps, cooking time, etc.) to the terminal.

[0068] 5. The device displays the received recipe to the user.

[0069] Real-time cooking support

[0070] 1. When a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and enters the question into the device.

[0071] 2. The device sends the captured photo and question to the server.

[0072] 3. The server analyzes the received photo and question and generates appropriate advice. For example, an AI algorithm determines the doneness of the chicken and generates advice such as "Continue cooking a little longer."

[0073] 4. The server sends the generated advice to the terminal.

[0074] 5. The device displays the advice to the user.

[0075] Educational support and cooking skill development

[0076] 1. The server analyzes data about the user's past cooking history and current skill level.

[0077] 2. The server generates cooking guidance based on the user's current skill level (e.g., how to make a sauce or adjust the heat).

[0078] 3. The server sends the generated guidance to the terminal.

[0079] 4. The device displays the received guidance to the user and provides assistance in improving cooking skills step by step.

[0080] As described above, the "SmartChef" system of the present invention combines image recognition technology, generative AI models, and real-time cooking support to maximize the use of ingredients users have at home and provide an efficient and enjoyable cooking experience, thereby reducing food waste and promoting healthy eating habits.

[0081] The processing flow will be explained below.

[0082] Step 1:

[0083] The user takes some ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[0084] Step 2:

[0085] The device preprocesses the captured photos (adjusting the resolution, converting the format, etc.) and sends them to the server.

[0086] Step 3:

[0087] The server applies an image recognition algorithm (e.g., YOLO, ResNet, etc.) to analyze the received image and identify the ingredients in the photo. Let's say the identified ingredients are tomatoes, chicken, and carrots.

[0088] Step 4:

[0089] The server generates a list of the identified ingredients and returns it to the terminal.

[0090] Step 5:

[0091] The user checks the list of ingredients displayed on the device and enters their preferences and allergy information (e.g., I don't like spicy food).

[0092] Step 6:

[0093] The terminal transmits the ingredient list and the user's preferences and allergy information to the server.

[0094] Step 7:

[0095] The server uses a generative AI model (e.g., GPT-4 or Fine-Tuned model) based on the received information to generate a customized recipe, such as a "Tomato and Chicken Stew" recipe.

[0096] Step 8:

[0097] The server sends the generated recipe (ingredients, steps, cooking time, etc.) back to the terminal.

[0098] Step 9:

[0099] The device displays the received recipe to the user.

[0100] Step 10:

[0101] If a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and types the question into the device.

[0102] Step 11:

[0103] The device sends the captured photo and the question to the server.

[0104] Step 12:

[0105] The server analyzes the received photos and questions and generates appropriate advice, such as "Please continue baking a little longer."

[0106] Step 13:

[0107] The server transmits the generated advice to the terminal.

[0108] Step 14:

[0109] The device displays the advice to the user.

[0110] Step 15:

[0111] The server analyzes data about the user's past cooking history and current skill level.

[0112] Step 16:

[0113] The server generates cooking guidance based on the user's current skill level (e.g., how to make a sauce or adjust the heat).

[0114] Step 17:

[0115] The server transmits the generated guidance to the terminal.

[0116] Step 18:

[0117] The device displays the received guidance to the user, providing assistance in improving cooking skills step by step.

[0118] In this way, the invention combines image recognition technology, generative AI models, and real-time cooking support to maximize the use of ingredients a user has on hand and provide an efficient and enjoyable cooking experience, reducing food waste and promoting healthy eating habits.

[0119] Example 1

[0120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0121] In modern households, it is often difficult to effectively utilize ingredients available in the refrigerator to prepare delicious and healthy meals. In particular, there is a need for systems that can suggest recipes based on ingredients, resolve cooking questions, and accommodate individual user preferences and allergies. However, conventional systems lack systems that meet these requirements, resulting in increased food waste and cooking stress for users. The present invention aims to solve the above problems and provide users with a convenient and effective cooking assistance system.

[0122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0123] In this invention, the server includes a means for a user to take images of ingredients owned by the user using a camera, a means for receiving the taken images and identifying the ingredients using image recognition technology, and a means for generating customized recipes using a generative AI model based on the identified ingredients and the user's preferences and allergy information, and providing the recipes to the user, thereby enabling the user to optimally utilize ingredients on hand and receive personalized recipes and cooking support.

[0124] "User" refers to the end user who uses the system to manage ingredients and cook.

[0125] A "photography device" is a device owned by a user for taking pictures of ingredients, and includes, for example, a smartphone camera.

[0126] "Image recognition technology" refers to algorithms and methods for automatically analyzing and identifying ingredients contained in a photographed image.

[0127] "Generative AI Model" refers to an artificial intelligence model used to generate customized recipes based on identified ingredients and user information.

[0128] A "customized recipe" is a recipe that includes individual cooking instructions and an ingredient list that are generated based on the ingredients a user has on hand and that takes into account the user's preferences and allergy information.

[0129] "Cooking Support" is a function that provides appropriate advice and solutions in real time to users when they have questions or difficulties while cooking.

[0130] "Guidance" refers to specific instruction and advice provided to improve a user's cooking skills.

[0131] "Ingredients" are food ingredients that are photographed by the user using a camera and recognized by the system.

[0132] "Preferences and allergy information" refers to information about a user's individual dietary preferences and allergies that the user enters in advance.

[0133] The system of the present invention is designed to maximize the use of ingredients on hand by the user and provide personalized recipes. This system consists of three main components: the user, the terminal, and the server, and their cooperation provides advanced cooking support.

[0134] Overview of the hardware and software used

[0135] 1. User:

[0136] Photography device: Use a device such as a smartphone camera to take pictures of ingredients you own.

[0137] Input device: Use a smartphone or tablet touch screen, keyboard, etc. to input preferences, allergy information, and questions during cooking.

[0138] 2. Device (smartphone, tablet, PC, etc.):

[0139] Image pre-processing software: Adjusts the resolution of received images, converts formats, and removes noise.

[0140] User interface: Displays ingredient lists, customizable recipes, and real-time advice.

[0141] 3. Server:

[0142] Image recognition algorithm: Using libraries such as TensorFlow, ingredients are identified from the received images.

[0143] Generative AI model: Utilizing GPT-3 and other models, it generates customized recipes based on identified ingredients, user preferences, and allergy information.

[0144] Data analysis unit: Analyzes the user's past cooking history and skill level and provides educational guidance.

[0145] Specific processing flow

[0146] Image recognition and food ingredient identification

[0147] 1. User:

[0148] Take some ingredients (e.g. tomatoes, chicken, carrots) out of the refrigerator and take a photo with your smartphone camera.

[0149] 2. Terminal:

[0150] Preprocess the photos (adjust resolution, convert format, etc.) and send them to the server.

[0151] 3. Server:

[0152] The received image is then processed using image recognition algorithms such as TensorFlow to identify ingredients, such as tomatoes, chicken, and carrots.

[0153] The identified ingredient list is returned to the terminal.

[0154] Customized recipe generation

[0155] 1. User:

[0156] Check the list of identified ingredients and enter your preferences and allergy information into the device (e.g., I don't like spicy food).

[0157] 2. Terminal:

[0158] A list of ingredients and information on preferences and allergies is sent to the server.

[0159] 3. Server:

[0160] A generative AI model (e.g., GPT-3) is used to generate a customized recipe, for example, "Chicken and Tomato Stew."

[0161] The generated recipe (ingredients, steps, cooking time, etc.) is sent back to the device.

[0162] 4. Terminal:

[0163] Display the received recipe to the user.

[0164] Real-time cooking support

[0165] 1. User:

[0166] If you have any questions while cooking (e.g., "Is this chicken cooked enough?"), take a photo of the cooking process, enter your question, and send it to the device.

[0167] 2. Terminal:

[0168] Send the question and image to the server.

[0169] 3. Server:

[0170] It uses AI algorithms to analyze the question and generate appropriate advice (e.g., "Please continue cooking a little longer").

[0171] The generated advice is sent to the terminal.

[0172] 4. Terminal:

[0173] Display advice to the user.

[0174] Educational support and cooking skill development

[0175] 1. Server:

[0176] Analyze data about your cooking history and current skill level.

[0177] Generate cooking guidance based on skill level (e.g., how to make a sauce or adjust the heat).

[0178] Send guidance to the device.

[0179] 2. Terminal:

[0180] The received guidance is displayed to the user, providing assistance in improving cooking skills step by step.

[0181] Specific examples

[0182] For example, if a user takes an image of a tomato, chicken, and carrot, the following prompt sentences are input into the generative AI model:

[0183] Example prompt sentence:

[0184] "Generate a non-spicy recipe using tomatoes, chicken, and carrots."

[0185] This prompt will cause the generative AI model to generate a recipe for "Tomato and Chicken Stew" and send it to the device.

[0186] As described above, by providing advanced cooking support through collaboration between the user, device, and server, users can make optimal use of the ingredients they have on hand and receive personalized recipes and cooking support.

[0187] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0188] Step 1:

[0189] The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a picture of the ingredients with the smartphone camera. The user then sends the image to the device via the "SmartChef" app. The input data is the image of the ingredients, and the output data is the image file sent to the device.

[0190] Step 2:

[0191] The terminal preprocesses the received image. Preprocessing includes adjusting the image resolution, format conversion, and noise removal, making the image suitable for analysis. The input data is the image file sent by the user, and the output data is the preprocessed image file.

[0192] Step 3:

[0193] The device sends the preprocessed image file to the server, which uses an image recognition algorithm such as TensorFlow to identify ingredients from the received image. The input data is the preprocessed image file, and the output data is a list of identified ingredients.

[0194] Step 4:

[0195] The server sends the identified ingredient list to the terminal. The terminal displays the ingredient list to the user. The user checks the displayed ingredient list, checks for missing information or errors, and corrects it as necessary. The input data is the ingredient list sent from the server, and the output data is the corrected ingredient list.

[0196] Step 5:

[0197] The user inputs their preferences and allergy information into the terminal. For example, they input "I don't like spicy food." The terminal then sends the input preference and allergy information along with the ingredient list to the server. The input data is the user's preference and allergy information and the ingredient list, and the output data is the information sent to the server.

[0198] Step 6:

[0199] The server generates a customized recipe using a generative AI model (e.g., GPT-3) based on the received information. The input data is the user's preferences, allergy information, and a list of ingredients, and the output data is the generated customized recipe. For example, a recipe called "Tomato and Chicken Stew" is generated.

[0200] Step 7:

[0201] The server sends the generated customized recipe to the terminal. The terminal displays the received recipe to the user. The input data is the generated customized recipe, and the output data is the recipe displayed on the terminal.

[0202] Step 8:

[0203] When a question arises during cooking (e.g., "Is this chicken cooked thoroughly?"), the user takes a photo of the chicken being cooked and enters the question into the terminal and sends it. The input data is the image of the chicken being cooked and the question, and the output data is the data sent to the terminal.

[0204] Step 9:

[0205] The device sends the captured photo and question to the server, which then uses an AI algorithm to analyze the image and question and generate appropriate advice. For example, the advice generated might be, "Please continue cooking a little longer." The input data is the image of the chicken and the question, and the output data is the generated advice.

[0206] Step 10:

[0207] The server sends the generated advice to the terminal. The terminal displays the received advice to the user. The input data is the generated advice, and the output data is the advice displayed on the terminal.

[0208] Step 11:

[0209] The server analyzes data related to the user's cooking history and current skill level and generates guidance for skill improvement. For example, guidance on "how to make a sauce and how to adjust the heat" is generated. The input data is the user's cooking history and skill level, and the output data is the generated guidance.

[0210] Step 12:

[0211] The server transmits the generated guidance to the terminal. The terminal displays the received guidance to the user, providing support for step-by-step improvement of cooking skills. The input data is the generated guidance, and the output data is the guidance displayed on the terminal.

[0212] (Application example 1)

[0213] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0214] In today's busy lifestyles, consumers are expected to prepare healthy, balanced meals while using their time efficiently. However, many difficulties and questions often arise during the process from purchasing ingredients to cooking. In particular, users feel anxious about selecting ingredients, and there is a lack of support that can address questions they have while cooking in real time. This can lead to food waste and make it difficult to prepare healthy meals. To solve these problems, the present invention aims to provide a system that supports the entire process from ingredient selection to cooking in a physical store, allowing users to select ingredients with confidence and cook effectively.

[0215] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0216] In this invention, the server includes means for a user to take images of ingredients owned by the user using a camera, means for receiving the taken images and identifying the ingredients using image recognition technology, means for providing the user with a customized recipe generated based on the identified ingredients, means for inputting the user's preferences and allergy information, means for adjusting the generation of the customized recipe taking into account the input preferences and allergy information, means for scanning ingredients in a physical store and providing an ingredient list and recipe on the spot, and means for resolving questions the user has about ingredient selection in real time. This allows the user to efficiently select ingredients in a physical store and receive real-time support for any questions or difficulties they may have about cooking even after purchase.

[0217] A "user" is a consumer who uses the system, inputs information related to ingredient selection and cooking, and receives feedback.

[0218] A "photography device" is a device owned by the user that includes a camera for taking pictures of ingredients and cooking conditions, and includes smartphones and tablets.

[0219] "Image recognition technology" is a technology that uses computer vision and machine learning algorithms to identify specific ingredients from photographed images.

[0220] A "customized recipe" is a personalized recipe created based on the user's ingredients, preferences, and allergy information.

[0221] "Preference and allergy information" refers to information about the user's food preferences and allergies, and is taken into consideration when generating customized recipes.

[0222] A "physical store" is a physical store where users directly purchase ingredients, such as a supermarket or grocery store.

[0223] "Scanning" is the process by which a user takes a picture of an ingredient in a physical store using a camera, allowing the system to identify the ingredient.

[0224] "Real-time support" is a feature that provides immediate advice and solutions to questions or difficulties users may encounter while selecting ingredients or cooking.

[0225] The "SmartShop" system supports the workflow from ingredient selection to cooking in a brick-and-mortar store. The system consists of a user-specific application (hereafter referred to as a terminal) and a server that performs image recognition and recipe generation.

[0226] System configuration

[0227] The components of this system are as follows:

[0228] 1. Terminal

[0229] Camera: Uses a smartphone or tablet camera to capture the ingredients the user selects in the store and the cooking process.

[0230] User Interface: Displays the ingredient list and customized recipe, and provides a means for users to input preferences and allergy information.

[0231] Communication function: Sends captured images and questions to a server and receives responses from the server.

[0232] 2. Server

[0233] Image Recognition: Uses computer vision and machine learning algorithms to analyze user-submitted images and identify ingredients.

[0234] Generative AI model: Generates customized recipes based on identified ingredients and user preferences and allergies.

[0235] Real-time support function: Generates and provides immediate advice and solutions to any questions or difficulties encountered while cooking.

[0236] Program processing explanation

[0237] Hardware and Software

[0238] Smartphone / Tablet (e.g. iPhone, Samsung Galaxy, etc.): Serves as a user interface and capture device.

[0239] Server: Use a cloud-based server such as Amazon Web Services (AWS) or Google Cloud Platform (GCP).

[0240] Python + Flask: Used to build the backend API.

[0241] PIL (Python Imaging Library) and numpy: Used for image and data processing.

[0242] Computer vision technology: OpenCV and TensorFlow are used as algorithms for food ingredient recognition.

[0243] Generative AI models: Use generative AI, such as OpenAI's GPT-3, to generate customized recipes.

[0244] Data processing and calculation

[0245] The device takes pictures of ingredients and cooking conditions using the smartphone camera and sends the image data to the server, where it is pre-processed by adjusting the resolution and converting the format.

[0246] The server analyzes the received image using computer vision technology to identify the ingredients, using neural network models for ingredient identification (e.g., ResNet, YOLO, etc.).

[0247] The generative AI model takes the identified ingredient list and the user's preferences and allergy information as inputs to generate customized recipes, providing recipes tailored to the user's needs.

[0248] The real-time support function analyzes captured images and questions that users may have while cooking, and generates appropriate advice using a generative AI model, which is then sent back to the device.

[0249] Specific examples

[0250] Example 1: Use in a physical store

[0251] A user scans "tomatoes, chicken, and carrots" in a physical store using their smartphone. Based on this information and the user's pre-entered preferences (such as not liking spicy food), the system provides a recipe for "tomato and chicken stew." If the user has questions about how to check the freshness of tomatoes at the time of purchase, the server immediately returns advice.

[0252] Example prompt:

[0253] Ingredients: Tomato, chicken, carrot. Preferences: Don't like spicy food. Allergies: None. Please recommend some recipes using these ingredients.

[0254] This allows users to smoothly select ingredients and cook them, reducing food waste and providing an efficient and enjoyable cooking experience.

[0255] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0256] Step 1:

[0257] The user takes a photo of the ingredients they plan to purchase in a physical store using the smartphone camera. The user then takes a picture of the ingredients and saves the image on the device through the application. The input is the photographed image of the ingredients, and the output is preprocessed image data.

[0258] Step 2:

[0259] The device preprocesses the captured images and sends them to the server. Preprocessing includes image resolution adjustment, format conversion, noise removal, etc. The input is the captured raw image data, and the output is the preprocessed image data.

[0260] Step 3:

[0261] The server analyzes the received image data and identifies the ingredients using image recognition technology. Specifically, it uses computer vision algorithms (e.g., YOLO, ResNet) to identify the type of ingredients. The input is the preprocessed image data, and the output is a list of identified ingredients.

[0262] Step 4:

[0263] The server generates a list of identified ingredients and returns it to the device. The user can then check the list on the device. The input is the ingredient information identified by image recognition, and the output is the ingredient list.

[0264] Step 5:

[0265] The user inputs their preferences and allergy information into the terminal. The input is the user's preferences and allergies, and the output is the updated information.

[0266] Step 6:

[0267] The terminal transmits the ingredient list and the user's preferences and allergy information to the server. The input is the user input information and the specified ingredient list, and the output is the data sent to the server.

[0268] Step 7:

[0269] The server generates a customized recipe using a generative AI model based on the received information. The AI ​​model uses OpenAI's GPT-3 and other models to automatically generate recipes that take into account the identified ingredients and the user's preferences and allergy information. The input is the identified ingredient list and the user's preference data, and the output is a customized recipe.

[0270] Step 8:

[0271] The server returns the generated customized recipe to the terminal, which then displays the received recipe to the user. The input is the generated recipe information, and the output is the recipe displayed on the user interface.

[0272] Step 9:

[0273] If a user has a question or difficulty while cooking, they can take a photo and input the question into the device. The input is the cooking question and related images, and the output is the question data stored on the device.

[0274] Step 10:

[0275] The terminal sends the user's question and related images to the server. The input is the question data from the user and the related images, and the output is the data sent to the server.

[0276] Step 11:

[0277] The server analyzes the received questions and images and generates appropriate advice and solutions. It again uses a generative AI model to derive a specific solution to the question or problem. The input is the user's question data and related images, and the output is the generated advice.

[0278] Step 12:

[0279] The server sends the generated advice back to the terminal, which displays it to the user. The input is the advice data, and the output is the advice displayed in the user interface.

[0280] This process allows users to efficiently select ingredients in-store and receive real-time support for any cooking questions or difficulties after purchase.

[0281] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0282] System Overview

[0283] The system of the present invention, which aims to maximize the use of ingredients on hand and provide personalized recipes, integrates image recognition technology, generative AI models, real-time cooking support, and an emotion engine that recognizes user emotions.

[0284] System Components

[0285] 1. Users

[0286] Use a camera (such as a smartphone camera) to take a picture of the ingredients you own.

[0287] The captured image is sent to the server via the terminal.

[0288] Take a photo of any questions or problems that arise while cooking and enter them into the device as a question.

[0289] Enter any necessary preferences, allergies, and emotional information into the device.

[0290] The system provides cooking guidance based on skill level and relaxing recipes.

[0291] 2. Terminal

[0292] Images taken by the user are preprocessed and sent to the server.

[0293] Display received ingredient lists and customized recipes to users.

[0294] It receives questions and related images from users while they are cooking and sends them to the server.

[0295] Display advice and guidance received from the server.

[0296] Collects user emotion data and sends it to the server.

[0297] Providing cooking support based on the user's emotional state.

[0298] 3. Server

[0299] An image recognition algorithm is used to identify ingredients from the received image.

[0300] A customized recipe is generated based on the identified ingredient list.

[0301] Analyzes user questions and images to generate appropriate advice and solutions.

[0302] The generated ingredient list, customized recipe, and advice are sent back to the device.

[0303] The contents of the customized recipe are adjusted taking into account the user's preferences and allergy information.

[0304] It uses an emotion engine to analyze the user's emotional state and provides cooking support and recipes based on the results.

[0305] Program processing and specific examples

[0306] Image recognition and food ingredient identification

[0307] 1. The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo with their smartphone camera.

[0308] 2. The device preprocesses the photo (adjusts resolution, converts format, etc.) and sends it to the server.

[0309] 3. The server applies an image recognition algorithm (e.g., YOLO, ResNet, etc.) to analyze the submitted image and identify the ingredients in the photo. Let's say the identified ingredients are tomatoes, chicken, and carrots.

[0310] 4. The server generates a list of the identified ingredients and returns it to the terminal.

[0311] Customized recipe generation

[0312] 1. The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[0313] 2. The device sends the ingredient list and the user's preferences and allergy information to the server.

[0314] 3. The server uses a generative AI model (e.g., GPT-4 or Fine-Tuned model) based on the received information to generate a customized recipe. For example, a recipe for "Tomato and Chicken Stew" using tomatoes and chicken is created.

[0315] 4. The server returns the generated recipe (ingredients, steps, cooking time, etc.) to the terminal.

[0316] 5. The device displays the received recipe to the user.

[0317] Real-time cooking support

[0318] 1. When a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and enters the question into the device.

[0319] 2. The device sends the captured photo and question to the server.

[0320] 3. The server analyzes the received photo and question and generates appropriate advice, such as "Please continue baking a little longer."

[0321] 4. The server sends the generated advice to the terminal.

[0322] 5. The device displays the advice to the user.

[0323] Supported by an emotional engine

[0324] 1. The user inputs emotional information (e.g., feeling stressed or tired) into the device.

[0325] 2. The device sends the user's emotional information to the server.

[0326] 3. The server uses an emotion engine to analyze the user's emotional state and provide appropriate support based on that state. For example, a user feeling stressed might be suggested a recipe for a relaxing herbal tea.

[0327] 4. The server sends the generated cooking instructions and recipes back to the device.

[0328] 5. The device displays the received support and recipes to the user.

[0329] As described above, the system of the present invention combines image recognition technology, generative AI models, real-time cooking support, and an emotion engine to maximize the use of ingredients in the home and provide an efficient and enjoyable cooking experience, thereby reducing food waste and promoting healthy eating habits.

[0330] The processing flow will be explained below.

[0331] Image recognition and food ingredient identification

[0332] Step 1:

[0333] The user takes some ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[0334] Step 2:

[0335] The device pre-processes this photo (adjusting resolution, converting format, etc.) and sends it to the server.

[0336] Step 3:

[0337] The server analyzes the received image using an image recognition algorithm (e.g., YOLO, ResNet, etc.) and identifies the ingredients in the photo. Let's say the identified ingredients are tomatoes, chicken, and carrots.

[0338] Step 4:

[0339] The server generates a list of the identified ingredients and returns it to the terminal.

[0340] Process for generating customized recipes

[0341] Step 5:

[0342] The user checks the list of ingredients displayed on the device and enters their preferences and allergy information (e.g., I don't like spicy food).

[0343] Step 6:

[0344] The terminal sends a request to the server along with a list of ingredients based on the entered preferences and allergy information.

[0345] Step 7:

[0346] The server uses a generative AI model (e.g., GPT-4 or Fine-Tuned model) based on the received information to generate a customized recipe, such as a "Tomato and Chicken Stew" recipe using tomatoes and chicken.

[0347] Step 8:

[0348] The server returns the generated customized recipe (ingredients, steps, cooking time, etc.) to the terminal.

[0349] Step 9:

[0350] The device displays the received recipe to the user.

[0351] Real-time cooking support processing

[0352] Step 10:

[0353] If a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and types the question into the device.

[0354] Step 11:

[0355] The device sends the captured photo and the question to the server.

[0356] Step 12:

[0357] The server analyzes the received photos and questions and generates appropriate advice, such as "Please continue baking a little longer."

[0358] Step 13:

[0359] The server transmits the generated advice to the terminal.

[0360] Step 14:

[0361] The device displays the advice to the user.

[0362] Support processing with emotion engine

[0363] Step 15:

[0364] The user inputs emotional information (e.g., feeling stressed or tired) into the device.

[0365] Step 16:

[0366] The terminal transmits the user's emotion information to the server.

[0367] Step 17:

[0368] The server uses an emotion engine to analyze the user's emotional state and provide appropriate support based on that state. For example, a user feeling stressed might be suggested a recipe for a relaxing herbal tea.

[0369] Step 18:

[0370] The server returns the generated cooking instructions and recipes to the terminal.

[0371] Step 19:

[0372] The device displays the received support and recipes to the user.

[0373] In this way, the present invention performs a series of processes to maximize the use of ingredients available to the user and provide an efficient and enjoyable cooking experience. By combining image recognition technology, generative AI models, real-time cooking support, and an emotion engine, it is possible to provide appropriate support and recipe suggestions based on the user's emotional state. This reduces food waste and promotes healthy eating habits.

[0374] Example 2

[0375] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0376] Conventional cooking support systems make it difficult for users to efficiently utilize the individual ingredients they own, and also make it difficult to receive immediate and appropriate support for questions or difficulties that arise during cooking. Furthermore, they are unable to adequately respond to users' emotional state or individual preferences, and are therefore unable to increase satisfaction with cooking. This has led to problems such as wasted ingredients, stress during the cooking process, and a lack of recipes tailored to individual needs.

[0377] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0378] In this invention, the server includes a means for analyzing images using an image recognition algorithm to identify ingredients, a means for generating customized recipes based on the ingredients identified by the generative AI model, and a means for analyzing the user's emotional information using an emotion engine, thereby enabling efficient use of ingredients, real-time cooking support, and recipe provision according to the user's emotional state and preferences.

[0379] "User" refers to an individual who uses the system to receive support in managing ingredients and cooking.

[0380] "Photography device" refers to a device such as a camera or smartphone used to take an image of an ingredient.

[0381] "Image pre-processing" refers to processing such as adjusting the resolution and converting the format of a captured image.

[0382] "Image recognition algorithms" refers to machine learning models and techniques (e.g., YOLO, ResNet) used to identify objects in images.

[0383] "Ingredients" refers to food owned or purchased by the User that is used as an ingredient in cooking.

[0384] "Generative AI Model" refers to an artificial intelligence model (e.g., GPT-4) that generates customized recipes based on identified ingredients.

[0385] "Customized Recipe" refers to a personalized cooking guide generated based on identified ingredients and taking into account the user's preferences and allergy information.

[0386] "Emotion engine" refers to technology that analyzes the user's emotional information and provides appropriate cooking support and recipes based on that state.

[0387] "Real-time cooking support" refers to the function that provides immediate and appropriate advice and solutions to questions or problems that arise while cooking.

[0388] "Preference and Allergy Information" means information related to individual dietary preferences and allergies that a User enters into the System.

[0389] The system of the present invention is designed to maximize the use of ingredients available to users and provide personalized recipes. The system integrates image recognition technology, generative AI models, real-time cooking support functions, and an emotion engine that recognizes user emotions.

[0390] System configuration

[0391] user

[0392] Users take pictures of ingredients they own using a camera such as a smartphone and send them to the device. A smartphone camera is a common camera. If a user has a question or problem while cooking, they can take a picture of the relevant item and enter it as a question into the device. In addition, users can enter information about their preferences, allergies, and emotions into the device.

[0393] Terminal

[0394] The device preprocesses the images taken by the user and sends them to the server. Preprocessing includes adjusting the resolution and converting the format (e.g., from JPEG to PNG). The device displays the received ingredient list and customized recipe, and sends questions and related images from the user while cooking to the server. The device also displays advice and guidance received from the server. Furthermore, the device collects the user's emotional data and sends it to the server.

[0395] server

[0396] The server uses an image recognition algorithm (e.g., YOLO, ResNet) to analyze the image sent by the user and identify the ingredients. Based on the identified ingredient list, the server generates a customized recipe using a generative AI model (e.g., GPT-4). The generated recipe includes ingredients, steps, cooking time, etc. and is sent to the device. The server also analyzes the user's questions and images, generates appropriate advice, and sends it to the device. Furthermore, the server uses an emotion engine to analyze the user's emotional state and provides cooking support and recipes based on the results.

[0397] Specific examples

[0398] Image recognition and food ingredient identification

[0399] 1. The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo with their smartphone camera.

[0400] 2. The device preprocesses the captured photo (e.g., adjusts resolution, converts format) and sends it to the server.

[0401] 3. The server applies an image recognition algorithm to identify the ingredients in the photo. Let's say the ingredients are tomatoes, chicken, and carrots.

[0402] 4. The server generates a list of the identified ingredients and returns it to the terminal.

[0403] Customized recipe generation

[0404] 1. The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[0405] 2. The device sends the ingredient list and the user's preferences and allergy information to the server.

[0406] 3. The server uses the generative AI model based on the received information to generate a customized recipe, for example, a "Chicken and Tomato Stew" recipe.

[0407] 4. The server sends the generated recipe (ingredients, steps, cooking time, etc.) to the terminal.

[0408] 5. The device displays the received recipe to the user.

[0409] Real-time cooking support

[0410] 1. When a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and enters the question into the device.

[0411] 2. The device sends the captured photo and question to the server.

[0412] 3. The server analyzes the received photo and question and generates appropriate advice, such as "Please continue baking a little longer."

[0413] 4. The server sends the generated advice to the terminal.

[0414] 5. The device displays the advice to the user.

[0415] Supported by an emotional engine

[0416] 1. The user inputs emotional information (e.g., feeling stressed or tired) into the device.

[0417] 2. The device sends the user's emotional information to the server.

[0418] 3. The server uses an emotion engine to analyze the user's emotional state and provide appropriate support based on that state. For example, a user feeling stressed might be suggested a recipe for a relaxing herbal tea.

[0419] 4. The server sends the generated cooking instructions and recipes back to the device.

[0420] 5. The device displays the received support and recipes to the user.

[0421] Prompt Sentence Examples

[0422] 1. Image recognition prompt:

[0423] "Please identify the ingredients in this image"

[0424] 2. Recipe generation prompt:

[0425] "Please suggest some non-spicy recipes using these ingredients."

[0426] 3. Cooking support prompt:

[0427] "Please judge whether the chicken in this photo is cooked thoroughly."

[0428] 4. Emotion Engine Support Prompt:

[0429] "Please suggest recipes that will help users relax when they are feeling stressed."

[0430] By using this system, users can use the ingredients they have on hand without waste and enjoy healthy and delicious meals.

[0431] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0432] Step 1:

[0433] The user takes the ingredients they plan to use (e.g., tomatoes, chicken, carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[0434] Input: Images of ingredients in the refrigerator

[0435] Output: Photographed food image (JPEG format)

[0436] Specific actions: Open the camera app on your smartphone, arrange the ingredients, take a photo, and press the shutter button with a "click!"

[0437] Step 2:

[0438] The device pre-processes the captured image, which includes adjusting the resolution and converting the format from JPEG to PNG.

[0439] Input: Photographed food image (JPEG format)

[0440] Output: Preprocessed food image (PNG format)

[0441] Specific operation: Run a program to adjust the resolution and convert the file format to PNG. The resolution adjustment is, for example, to make the width and height 800 pixels each.

[0442] Step 3:

[0443] The terminal sends the preprocessed image to the server and generates a prompt to start the image recognition algorithm.

[0444] Input: Preprocessed food image (PNG format)

[0445] Output: Image data and prompt text are sent to the server

[0446] Specific operation: The preprocessed image is sent to the server via an HTTP request, and a prompt message is sent saying, "Please identify the ingredients from this image."

[0447] Step 4:

[0448] The server applies an image recognition algorithm (e.g., YOLO, ResNet) to analyze the received image.

[0449] Input: Preprocessed food image, prompt

[0450] Output: List of recognized ingredients (e.g., tomato, chicken, carrot)

[0451] What it does: Runs the YOLO algorithm to detect objects in an image and generate an ingredients list. YOLO identifies tomatoes, chicken, and carrots in the image.

[0452] Step 5:

[0453] The server generates a customized recipe using a generative AI model (e.g., GPT-4) based on the identified ingredient list.

[0454] Input: List of recognized ingredients (e.g., tomato, chicken, carrot)

[0455] Output: Customized recipe (ingredients, steps, cooking time, etc.)

[0456] Specific behavior: GPT-4 is fed a list of ingredients and generates a recipe using the prompt "Please suggest a non-spicy recipe using these ingredients." The recipe "Tomato and Chicken Stew" is generated.

[0457] Step 6:

[0458] The server transmits the generated recipe to the terminal.

[0459] Input: Generated custom recipe

[0460] Output: Customized recipe data is sent to the device

[0461] What it does: It uses HTTP push notifications to send the generated recipe information to the device, including the ingredients list, instructions, cooking time, etc.

[0462] Step 7:

[0463] The device displays the received recipe to the user.

[0464] Input: Customized recipe data

[0465] Output: The recipe is displayed on the smartphone screen.

[0466] Specific Behavior: Runs a screen display program to format and display the received recipe information to the user. The UI includes the ingredient list, instructions, and cooking time.

[0467] Step 8:

[0468] If a user has a question while cooking, they can enter it, take a picture of the relevant information, and send it to the device.

[0469] Input: Cooking question or related image

[0470] Output: Questions and image data are sent from the device to the server.

[0471] How it works: You enter your question into your smartphone and take a photo of the related cooking state. The device then sends these to the server.

[0472] Step 9:

[0473] The server analyzes the received question and image and generates appropriate advice.

[0474] Input: Question and associated image data

[0475] Output: Advice (e.g. "Please continue to cook a little longer")

[0476] Specific operation: Analyzes the image using an image analysis algorithm and generates advice based on the question. Appropriate cooking advice is generated in text format.

[0477] Step 10:

[0478] The server transmits the generated advice to the terminal.

[0479] Input: Advice content

[0480] Output: Advice data is sent to the terminal

[0481] Specific operation: Advice information is sent to the device using HTTP push notifications.

[0482] Step 11:

[0483] The device displays the received advice to the user.

[0484] Input: Advice data

[0485] Output: Advice content is displayed on the smartphone screen

[0486] Specific behavior: Run the screen display program, format the received advice information, and display it to the user. The UI includes the advice text.

[0487] Step 12:

[0488] The user inputs their emotional state (e.g., feeling stressed) into the terminal.

[0489] Input: Emotion information

[0490] Output: Emotional information data is sent from the device to the server.

[0491] Specific actions: Enter your emotional state using the multiple choice input form and press the submit button.

[0492] Step 13:

[0493] The server analyzes the emotional information and generates cooking support and relaxing recipes according to the state of the user.

[0494] Input: Emotional information data

[0495] Output: Cooking support and relaxing recipes

[0496] Specific behavior: Activates the emotion engine to analyze the user's emotional state. Based on the analysis results, it generates a relaxing recipe (e.g., "How to make herbal tea").

[0497] Step 14:

[0498] The server transmits the generated cooking instructions and recipes to the terminal.

[0499] Input: Cooking support and relaxing recipes

[0500] Output: Support and recipe data sent to device

[0501] Specific operation: Send cooking support information to the device using HTTP push notifications.

[0502] Step 15:

[0503] The device displays the received support and recipes to the user.

[0504] Input: Cooking support and relaxation recipe data

[0505] Output: Support and recipe information displayed on the smartphone screen

[0506] Specific behavior: Executes the screen display program, formats the received support and recipe information, and displays it to the user. The UI includes support text and relaxation recipe text.

[0507] Through these processing steps, users can receive personalized recipes and real-time cooking support while making the most of the ingredients they have on hand. The system provides efficient ingredient utilization and a comfortable cooking experience.

[0508] (Application example 2)

[0509] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0510] Conventional cooking assistance systems are limited to providing recipes based on the ingredients a user has on hand, and are unable to provide real-time support during cooking or cooking assistance that takes into account the user's emotional state. Furthermore, they are unable to generate recipes that fully reflect the user's preferences and allergies. This has led to problems such as increased stress due to the difficulty users experience while cooking and a lack of suitable recipes.

[0511] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to take images of ingredients owned by the user using a camera, means for receiving the taken images and identifying the ingredients using image recognition technology, means for providing the user with a customized recipe generated based on the identified ingredients, means for generating a customized recipe based on the ingredient list and user information using a generative AI model, and means for inputting the user's emotional information and automatically adjusting the cooking support content provided using an emotion analysis engine. This enables more appropriate and personalized recipes and real-time cooking support that reflect the user's individual needs and preferences, as well as cooking support that responds to the user's emotional state.

[0512] A "photography device" is a device used to take pictures of ingredients owned by a user, such as a smartphone or camera.

[0513] "Image recognition technology" refers to algorithms and techniques for identifying and classifying specific objects or ingredients from photographed images, and includes models such as YOLO and ResNet.

[0514] A "generative AI model" is an artificial intelligence model that generates new text or recipes based on user input or identified data, and examples include GPT-4.

[0515] An "emotion analysis engine" is a technology that analyzes the emotional information entered by the user and provides appropriate cooking support and recipe adjustments based on the results.

[0516] A "customized recipe" is a recipe that meets a user's individual needs and is generated based on identified ingredients, the user's preferences, allergy information, and other individual information.

[0517] "Real-time cooking support" is a service that provides instant solutions and advice to users for any questions or difficulties they may have while cooking.

[0518] "User Information" means information about an individual user, such as preferences, allergy information, and emotional state, that is necessary to provide customized recipes and cooking support.

[0519] The "ingredient list" is a list of ingredients owned by the user, identified using image recognition technology.

[0520] The system embodying this invention integrates image recognition technology, generative AI models, real-time cooking support, and a sentiment analysis engine to recognize user emotions, in order to maximize the use of ingredients on hand and provide personalized recipes.

[0521] System Components

[0522] 1. Users

[0523] Use a camera (such as a smartphone camera) to take a picture of the ingredients you own.

[0524] The captured image is sent to the server via the terminal.

[0525] Take a photo of any questions or problems that arise while cooking and enter them into the device as a question.

[0526] Emotional information, such as stress level or relaxation level, is also entered into the device.

[0527] 2. Terminal

[0528] Images taken by the user are preprocessed and sent to the server.

[0529] Display received ingredient lists and customized recipes to users.

[0530] Collects user emotion data and sends it to the server.

[0531] 3. Server

[0532] Using an image recognition algorithm (e.g., YOLO, ResNet), ingredients are identified from the received image.

[0533] Generate a customized recipe using a generative AI model (e.g., GPT-4) based on the identified list of ingredients.

[0534] An emotion analysis engine that analyzes the user's emotional state is used to adjust the generated recipes and cooking support content.

[0535] Hardware and software used

[0536] Camera: Smartphone camera

[0537] Device: Smartphone or tablet

[0538] Server-side software:

[0539] Image recognition algorithms (e.g., YOLO, ResNet)

[0540] Generative AI models (e.g., GPT-4)

[0541] Sentiment Analysis Engine

[0542] Specific examples

[0543] 1. Image Recognition and Food Identification

[0544] The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[0545] The device preprocesses the photo (adjusting resolution, converting format, etc.) and sends it to the server.

[0546] The server applies an image recognition algorithm to identify the ingredients in the photo, which may be tomatoes, chicken, and carrots.

[0547] 2. Customized recipe generation

[0548] The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[0549] The device sends the ingredient list and the user's preferences and allergy information to the server.

[0550] The server uses the generative AI model to generate customized recipes, such as a "Tomato and Chicken Stew" recipe using tomatoes and chicken.

[0551] An example of a prompt sentence that will generate a recipe is:

[0552] Ingredients: Tomato, Chicken, Carrot

[0553] User Info: {"preferences": ["I hate spicy food"], "allergies": ["nuts"], "emotion_text": "I'm stressed"}

[0554] Recipe:

[0555] 3. Real-time cooking support

[0556] If a user has a question while cooking (e.g., "Is this chicken cooked enough?"), they take a photo of the cooked chicken and type the question into the device.

[0557] The device sends the captured photo and question to the server.

[0558] The server analyzes the received photo and question and generates appropriate advice, such as "Please continue baking a little longer."

[0559] 4. Support with sentiment analysis

[0560] The user's emotional information (e.g., feeling stressed) is input into the device.

[0561] The device transmits the user's emotional information to the server.

[0562] The server uses an emotion analysis engine to analyze the user's emotional state and provides cooking support and recipes based on the results.

[0563] The system allows users to make the most of the ingredients they have and easily obtain personalized recipes. It also responds to questions while cooking in real time and provides cooking support that adapts to their emotional state, making the cooking experience more efficient and enjoyable while reducing food waste.

[0564] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0565] Step 1:

[0566] Users take ingredients from the refrigerator and take a picture of them with their smartphone camera, which then stores the image on the device.

[0567] Step 2:

[0568] The device pre-processes the captured image, adjusts the resolution, and converts it to the appropriate format, then sends the pre-processed image to the server.

[0569] Step 3:

[0570] The server analyzes the received image using an image recognition algorithm (e.g., YOLO, ResNet), and generates a list of ingredients identified from the image and sends it back to the device.

[0571] Step 4:

[0572] Users can review the list of identified ingredients and enter their preferences and allergies into the device, including details such as an aversion to spicy food.

[0573] Step 5:

[0574] The device sends the input user information (ingredients list, preferences, allergy information) to the server, which uses a generative AI model (e.g., GPT-4) to generate a customized recipe based on the input information.

[0575] Step 6:

[0576] The server sends the created customized recipe to the terminal, which displays it to the user and prompts for confirmation.

[0577] Step 7:

[0578] When a user has a question while cooking, they can take a picture of the relevant food and enter it into the device along with their question.

[0579] Step 8:

[0580] The device sends the captured image and question to the server, which analyzes the received data, generates appropriate solutions or advice, and sends it back to the device.

[0581] Step 9:

[0582] The user inputs emotional information (e.g., feeling stressed) into the device, which then transmits this information to the server.

[0583] Step 10:

[0584] The server uses an emotion analysis engine to analyze the user's emotional state, and based on the results, adjusts cooking support and recipe content and sends it back to the device.

[0585] Step 11:

[0586] The device displays the adjusted recipe and cooking support information to the user, who then proceeds with cooking based on this information.

[0587] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0588] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0589] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0590] [Second embodiment]

[0591] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0592] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0593] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0594] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0595] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0596] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0597] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0598] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0599] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0600] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0601] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0602] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0603] System Overview

[0604] The "SmartChef" system of the present invention is a system that maximizes the use of ingredients on hand and provides personalized recipes. The system integrates image recognition technology, generative AI models, and real-time cooking support functions.

[0605] System Components

[0606] 1. Users

[0607] Use a camera (such as a smartphone camera) to take a picture of the ingredients you own.

[0608] The captured image is sent to the server via the terminal.

[0609] Take a photo of any questions or problems that arise while cooking and enter them into the device as a question.

[0610] Enter any necessary preferences or allergy information into the device.

[0611] 2. Terminal

[0612] Images taken by the user are preprocessed and sent to the server.

[0613] Display received ingredient lists and customized recipes to users.

[0614] It receives questions and related images from users while they are cooking and sends them to the server.

[0615] Display advice and guidance received from the server.

[0616] 3. Server

[0617] An image recognition algorithm is used to identify ingredients from the received image.

[0618] A customized recipe is generated based on the identified ingredient list.

[0619] Analyzes user questions and images to generate appropriate advice and solutions.

[0620] The generated ingredient list, customized recipe, and advice are sent back to the device.

[0621] The contents of the customized recipe are adjusted taking into account the user's preferences and allergy information.

[0622] Specific processing flow

[0623] Image recognition and food ingredient identification

[0624] 1. The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo with their smartphone camera.

[0625] 2. The device preprocesses the photo (adjusts resolution, converts format, etc.) and sends it to the server.

[0626] 3. The server receives the submitted image and applies image recognition algorithms to identify the ingredients in the photo, such as tomatoes, chicken, and carrots.

[0627] 4. The server generates a list of the identified ingredients and returns it to the terminal.

[0628] Customized recipe generation

[0629] 1. The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[0630] 2. The terminal sends the ingredient list and the user's preferences and allergy information to the server.

[0631] 3. The server uses the generative AI model based on the received information to generate a customized recipe, such as a "Tomato and Chicken Stew" recipe.

[0632] 4. The server returns the generated recipe (ingredients, steps, cooking time, etc.) to the terminal.

[0633] 5. The device displays the received recipe to the user.

[0634] Real-time cooking support

[0635] 1. When a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and enters the question into the device.

[0636] 2. The device sends the captured photo and question to the server.

[0637] 3. The server analyzes the received photo and question and generates appropriate advice. For example, an AI algorithm determines the doneness of the chicken and generates advice such as "Continue cooking a little longer."

[0638] 4. The server sends the generated advice to the terminal.

[0639] 5. The device displays the advice to the user.

[0640] Educational support and cooking skill development

[0641] 1. The server analyzes data about the user's past cooking history and current skill level.

[0642] 2. The server generates cooking guidance based on the user's current skill level (e.g., how to make a sauce or adjust the heat).

[0643] 3. The server sends the generated guidance to the terminal.

[0644] 4. The device displays the received guidance to the user and provides assistance in improving cooking skills step by step.

[0645] As described above, the "SmartChef" system of the present invention combines image recognition technology, generative AI models, and real-time cooking support to maximize the use of ingredients users have at home and provide an efficient and enjoyable cooking experience, thereby reducing food waste and promoting healthy eating habits.

[0646] The processing flow will be explained below.

[0647] Step 1:

[0648] The user takes some ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[0649] Step 2:

[0650] The device preprocesses the captured photos (adjusting the resolution, converting the format, etc.) and sends them to the server.

[0651] Step 3:

[0652] The server applies an image recognition algorithm (e.g., YOLO, ResNet, etc.) to analyze the received image and identify the ingredients in the photo. Let's say the identified ingredients are tomatoes, chicken, and carrots.

[0653] Step 4:

[0654] The server generates a list of the identified ingredients and returns it to the terminal.

[0655] Step 5:

[0656] The user checks the list of ingredients displayed on the device and enters their preferences and allergy information (e.g., I don't like spicy food).

[0657] Step 6:

[0658] The terminal transmits the ingredient list and the user's preferences and allergy information to the server.

[0659] Step 7:

[0660] The server uses a generative AI model (e.g., GPT-4 or Fine-Tuned model) based on the received information to generate a customized recipe, such as a "Tomato and Chicken Stew" recipe.

[0661] Step 8:

[0662] The server sends the generated recipe (ingredients, steps, cooking time, etc.) back to the terminal.

[0663] Step 9:

[0664] The device displays the received recipe to the user.

[0665] Step 10:

[0666] If a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and types the question into the device.

[0667] Step 11:

[0668] The device sends the captured photo and the question to the server.

[0669] Step 12:

[0670] The server analyzes the received photos and questions and generates appropriate advice, such as "Please continue baking a little longer."

[0671] Step 13:

[0672] The server transmits the generated advice to the terminal.

[0673] Step 14:

[0674] The device displays the advice to the user.

[0675] Step 15:

[0676] The server analyzes data about the user's past cooking history and current skill level.

[0677] Step 16:

[0678] The server generates cooking guidance based on the user's current skill level (e.g., how to make a sauce or adjust the heat).

[0679] Step 17:

[0680] The server transmits the generated guidance to the terminal.

[0681] Step 18:

[0682] The device displays the received guidance to the user, providing assistance in improving cooking skills step by step.

[0683] In this way, the invention combines image recognition technology, generative AI models, and real-time cooking support to maximize the use of ingredients a user has on hand and provide an efficient and enjoyable cooking experience, reducing food waste and promoting healthy eating habits.

[0684] Example 1

[0685] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0686] In modern households, it is often difficult to effectively utilize ingredients available in the refrigerator to prepare delicious and healthy meals. In particular, there is a need for systems that can suggest recipes based on ingredients, resolve cooking questions, and accommodate individual user preferences and allergies. However, conventional systems lack systems that meet these requirements, resulting in increased food waste and cooking stress for users. The present invention aims to solve the above problems and provide users with a convenient and effective cooking assistance system.

[0687] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0688] In this invention, the server includes a means for a user to take images of ingredients owned by the user using a camera, a means for receiving the taken images and identifying the ingredients using image recognition technology, and a means for generating customized recipes using a generative AI model based on the identified ingredients and the user's preferences and allergy information, and providing the recipes to the user, thereby enabling the user to optimally utilize ingredients on hand and receive personalized recipes and cooking support.

[0689] "User" refers to the end user who uses the system to manage ingredients and cook.

[0690] A "photography device" is a device owned by a user for taking pictures of ingredients, and includes, for example, a smartphone camera.

[0691] "Image recognition technology" refers to algorithms and methods for automatically analyzing and identifying ingredients contained in a photographed image.

[0692] "Generative AI Model" refers to an artificial intelligence model used to generate customized recipes based on identified ingredients and user information.

[0693] A "customized recipe" is a recipe that includes individual cooking instructions and an ingredient list that are generated based on the ingredients a user has on hand and that takes into account the user's preferences and allergy information.

[0694] "Cooking Support" is a function that provides appropriate advice and solutions in real time to users when they have questions or difficulties while cooking.

[0695] "Guidance" refers to specific instruction and advice provided to improve a user's cooking skills.

[0696] "Ingredients" are food ingredients that are photographed by the user using a camera and recognized by the system.

[0697] "Preferences and allergy information" refers to information about a user's individual dietary preferences and allergies that the user enters in advance.

[0698] The system of the present invention is designed to maximize the use of ingredients on hand by the user and provide personalized recipes. This system consists of three main components: the user, the terminal, and the server, and their cooperation provides advanced cooking support.

[0699] Overview of the hardware and software used

[0700] 1. User:

[0701] Photography device: Use a device such as a smartphone camera to take pictures of ingredients you own.

[0702] Input device: Use a smartphone or tablet touch screen, keyboard, etc. to input preferences, allergy information, and questions during cooking.

[0703] 2. Device (smartphone, tablet, PC, etc.):

[0704] Image pre-processing software: Adjusts the resolution of received images, converts formats, and removes noise.

[0705] User interface: Displays ingredient lists, customizable recipes, and real-time advice.

[0706] 3. Server:

[0707] Image recognition algorithm: Using libraries such as TensorFlow, ingredients are identified from the received images.

[0708] Generative AI model: Utilizing GPT-3 and other models, it generates customized recipes based on identified ingredients, user preferences, and allergy information.

[0709] Data analysis unit: Analyzes the user's past cooking history and skill level and provides educational guidance.

[0710] Specific processing flow

[0711] Image recognition and food ingredient identification

[0712] 1. User:

[0713] Take some ingredients (e.g. tomatoes, chicken, carrots) out of the refrigerator and take a photo with your smartphone camera.

[0714] 2. Terminal:

[0715] Preprocess the photos (adjust resolution, convert format, etc.) and send them to the server.

[0716] 3. Server:

[0717] The received image is then processed using image recognition algorithms such as TensorFlow to identify ingredients, such as tomatoes, chicken, and carrots.

[0718] The identified ingredient list is returned to the terminal.

[0719] Customized recipe generation

[0720] 1. User:

[0721] Check the list of identified ingredients and enter your preferences and allergy information into the device (e.g., I don't like spicy food).

[0722] 2. Terminal:

[0723] A list of ingredients and information on preferences and allergies is sent to the server.

[0724] 3. Server:

[0725] A generative AI model (e.g., GPT-3) is used to generate a customized recipe, for example, "Chicken and Tomato Stew."

[0726] The generated recipe (ingredients, steps, cooking time, etc.) is sent back to the device.

[0727] 4. Terminal:

[0728] Display the received recipe to the user.

[0729] Real-time cooking support

[0730] 1. User:

[0731] If you have any questions while cooking (e.g., "Is this chicken cooked enough?"), take a photo of the cooking process, enter your question, and send it to the device.

[0732] 2. Terminal:

[0733] Send the question and image to the server.

[0734] 3. Server:

[0735] It uses AI algorithms to analyze the question and generate appropriate advice (e.g., "Please continue cooking a little longer").

[0736] The generated advice is sent to the terminal.

[0737] 4. Terminal:

[0738] Display advice to the user.

[0739] Educational support and cooking skill development

[0740] 1. Server:

[0741] Analyze data about your cooking history and current skill level.

[0742] Generate cooking guidance based on skill level (e.g., how to make a sauce or adjust the heat).

[0743] Send guidance to the device.

[0744] 2. Terminal:

[0745] The received guidance is displayed to the user, providing assistance in improving cooking skills step by step.

[0746] Specific examples

[0747] For example, if a user takes an image of a tomato, chicken, and carrot, the following prompt sentences are input into the generative AI model:

[0748] Example prompt sentence:

[0749] "Generate a non-spicy recipe using tomatoes, chicken, and carrots."

[0750] This prompt will cause the generative AI model to generate a recipe for "Tomato and Chicken Stew" and send it to the device.

[0751] As described above, by providing advanced cooking support through collaboration between the user, device, and server, users can make optimal use of the ingredients they have on hand and receive personalized recipes and cooking support.

[0752] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0753] Step 1:

[0754] The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a picture of the ingredients with the smartphone camera. The user then sends the image to the device via the "SmartChef" app. The input data is the image of the ingredients, and the output data is the image file sent to the device.

[0755] Step 2:

[0756] The terminal preprocesses the received image. Preprocessing includes adjusting the image resolution, format conversion, and noise removal, making the image suitable for analysis. The input data is the image file sent by the user, and the output data is the preprocessed image file.

[0757] Step 3:

[0758] The device sends the preprocessed image file to the server, which uses an image recognition algorithm such as TensorFlow to identify ingredients from the received image. The input data is the preprocessed image file, and the output data is a list of identified ingredients.

[0759] Step 4:

[0760] The server sends the identified ingredient list to the terminal. The terminal displays the ingredient list to the user. The user checks the displayed ingredient list, checks for missing information or errors, and corrects it as necessary. The input data is the ingredient list sent from the server, and the output data is the corrected ingredient list.

[0761] Step 5:

[0762] The user inputs their preferences and allergy information into the terminal. For example, they input "I don't like spicy food." The terminal then sends the input preference and allergy information along with the ingredient list to the server. The input data is the user's preference and allergy information and the ingredient list, and the output data is the information sent to the server.

[0763] Step 6:

[0764] The server generates a customized recipe using a generative AI model (e.g., GPT-3) based on the received information. The input data is the user's preferences, allergy information, and a list of ingredients, and the output data is the generated customized recipe. For example, a recipe called "Tomato and Chicken Stew" is generated.

[0765] Step 7:

[0766] The server sends the generated customized recipe to the terminal. The terminal displays the received recipe to the user. The input data is the generated customized recipe, and the output data is the recipe displayed on the terminal.

[0767] Step 8:

[0768] When a question arises during cooking (e.g., "Is this chicken cooked thoroughly?"), the user takes a photo of the chicken being cooked and enters the question into the terminal and sends it. The input data is the image of the chicken being cooked and the question, and the output data is the data sent to the terminal.

[0769] Step 9:

[0770] The device sends the captured photo and question to the server, which then uses an AI algorithm to analyze the image and question and generate appropriate advice. For example, the advice generated might be, "Please continue cooking a little longer." The input data is the image of the chicken and the question, and the output data is the generated advice.

[0771] Step 10:

[0772] The server sends the generated advice to the terminal. The terminal displays the received advice to the user. The input data is the generated advice, and the output data is the advice displayed on the terminal.

[0773] Step 11:

[0774] The server analyzes data related to the user's cooking history and current skill level and generates guidance for skill improvement. For example, guidance on "how to make a sauce and how to adjust the heat" is generated. The input data is the user's cooking history and skill level, and the output data is the generated guidance.

[0775] Step 12:

[0776] The server transmits the generated guidance to the terminal. The terminal displays the received guidance to the user, providing support for step-by-step improvement of cooking skills. The input data is the generated guidance, and the output data is the guidance displayed on the terminal.

[0777] (Application example 1)

[0778] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0779] In today's busy lifestyles, consumers are expected to prepare healthy, balanced meals while using their time efficiently. However, many difficulties and questions often arise during the process from purchasing ingredients to cooking. In particular, users feel anxious about selecting ingredients, and there is a lack of support that can address questions they have while cooking in real time. This can lead to food waste and make it difficult to prepare healthy meals. To solve these problems, the present invention aims to provide a system that supports the entire process from ingredient selection to cooking in a physical store, allowing users to select ingredients with confidence and cook effectively.

[0780] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0781] In this invention, the server includes means for a user to take images of ingredients owned by the user using a camera, means for receiving the taken images and identifying the ingredients using image recognition technology, means for providing the user with a customized recipe generated based on the identified ingredients, means for inputting the user's preferences and allergy information, means for adjusting the generation of the customized recipe taking into account the input preferences and allergy information, means for scanning ingredients in a physical store and providing an ingredient list and recipe on the spot, and means for resolving questions the user has about ingredient selection in real time. This allows the user to efficiently select ingredients in a physical store and receive real-time support for any questions or difficulties they may have about cooking even after purchase.

[0782] A "user" is a consumer who uses the system, inputs information related to ingredient selection and cooking, and receives feedback.

[0783] A "photography device" is a device owned by the user that includes a camera for taking pictures of ingredients and cooking conditions, and includes smartphones and tablets.

[0784] "Image recognition technology" is a technology that uses computer vision and machine learning algorithms to identify specific ingredients from photographed images.

[0785] A "customized recipe" is a personalized recipe created based on the user's ingredients, preferences, and allergy information.

[0786] "Preference and allergy information" refers to information about the user's food preferences and allergies, and is taken into consideration when generating customized recipes.

[0787] A "physical store" is a physical store where users directly purchase ingredients, such as a supermarket or grocery store.

[0788] "Scanning" is the process by which a user takes a picture of an ingredient in a physical store using a camera, allowing the system to identify the ingredient.

[0789] "Real-time support" is a feature that provides immediate advice and solutions to questions or difficulties users may encounter while selecting ingredients or cooking.

[0790] The "SmartShop" system supports the workflow from ingredient selection to cooking in a brick-and-mortar store. The system consists of a user-specific application (hereafter referred to as a terminal) and a server that performs image recognition and recipe generation.

[0791] System configuration

[0792] The components of this system are as follows:

[0793] 1. Terminal

[0794] Camera: Uses a smartphone or tablet camera to capture the ingredients the user selects in the store and the cooking process.

[0795] User Interface: Displays the ingredient list and customized recipe, and provides a means for users to input preferences and allergy information.

[0796] Communication function: Sends captured images and questions to a server and receives responses from the server.

[0797] 2. Server

[0798] Image Recognition: Uses computer vision and machine learning algorithms to analyze user-submitted images and identify ingredients.

[0799] Generative AI model: Generates customized recipes based on identified ingredients and user preferences and allergies.

[0800] Real-time support function: Generates and provides immediate advice and solutions to any questions or difficulties encountered while cooking.

[0801] Program processing explanation

[0802] Hardware and Software

[0803] Smartphone / Tablet (e.g. iPhone, Samsung Galaxy, etc.): Serves as a user interface and capture device.

[0804] Server: Use a cloud-based server such as Amazon Web Services (AWS) or Google Cloud Platform (GCP).

[0805] Python + Flask: Used to build the backend API.

[0806] PIL (Python Imaging Library) and numpy: Used for image and data processing.

[0807] Computer vision technology: OpenCV and TensorFlow are used as algorithms for food ingredient recognition.

[0808] Generative AI models: Use generative AI, such as OpenAI's GPT-3, to generate customized recipes.

[0809] Data processing and calculation

[0810] The device takes pictures of ingredients and cooking conditions using the smartphone camera and sends the image data to the server, where it is pre-processed by adjusting the resolution and converting the format.

[0811] The server analyzes the received image using computer vision technology to identify the ingredients, using neural network models for ingredient identification (e.g., ResNet, YOLO, etc.).

[0812] The generative AI model takes the identified ingredient list and the user's preferences and allergy information as inputs to generate customized recipes, providing recipes tailored to the user's needs.

[0813] The real-time support function analyzes captured images and questions that users may have while cooking, and generates appropriate advice using a generative AI model, which is then sent back to the device.

[0814] Specific examples

[0815] Example 1: Use in a physical store

[0816] A user scans "tomatoes, chicken, and carrots" in a physical store using their smartphone. Based on this information and the user's pre-entered preferences (such as not liking spicy food), the system provides a recipe for "tomato and chicken stew." If the user has questions about how to check the freshness of tomatoes at the time of purchase, the server immediately returns advice.

[0817] Example prompt:

[0818] Ingredients: Tomato, chicken, carrot. Preferences: Don't like spicy food. Allergies: None. Please recommend some recipes using these ingredients.

[0819] This allows users to smoothly select ingredients and cook them, reducing food waste and providing an efficient and enjoyable cooking experience.

[0820] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0821] Step 1:

[0822] The user takes a photo of the ingredients they plan to purchase in a physical store using the smartphone camera. The user then takes a picture of the ingredients and saves the image on the device through the application. The input is the photographed image of the ingredients, and the output is preprocessed image data.

[0823] Step 2:

[0824] The device preprocesses the captured images and sends them to the server. Preprocessing includes image resolution adjustment, format conversion, noise removal, etc. The input is the captured raw image data, and the output is the preprocessed image data.

[0825] Step 3:

[0826] The server analyzes the received image data and identifies the ingredients using image recognition technology. Specifically, it uses computer vision algorithms (e.g., YOLO, ResNet) to identify the type of ingredients. The input is the preprocessed image data, and the output is a list of identified ingredients.

[0827] Step 4:

[0828] The server generates a list of identified ingredients and returns it to the device. The user can then check the list on the device. The input is the ingredient information identified by image recognition, and the output is the ingredient list.

[0829] Step 5:

[0830] The user inputs their preferences and allergy information into the terminal. The input is the user's preferences and allergies, and the output is the updated information.

[0831] Step 6:

[0832] The terminal transmits the ingredient list and the user's preferences and allergy information to the server. The input is the user input information and the specified ingredient list, and the output is the data sent to the server.

[0833] Step 7:

[0834] The server generates a customized recipe using a generative AI model based on the received information. The AI ​​model uses OpenAI's GPT-3 and other models to automatically generate recipes that take into account the identified ingredients and the user's preferences and allergy information. The input is the identified ingredient list and the user's preference data, and the output is a customized recipe.

[0835] Step 8:

[0836] The server returns the generated customized recipe to the terminal, which then displays the received recipe to the user. The input is the generated recipe information, and the output is the recipe displayed on the user interface.

[0837] Step 9:

[0838] If a user has a question or difficulty while cooking, they can take a photo and input the question into the device. The input is the cooking question and related images, and the output is the question data stored on the device.

[0839] Step 10:

[0840] The terminal sends the user's question and related images to the server. The input is the question data from the user and the related images, and the output is the data sent to the server.

[0841] Step 11:

[0842] The server analyzes the received questions and images and generates appropriate advice and solutions. It again uses a generative AI model to derive a specific solution to the question or problem. The input is the user's question data and related images, and the output is the generated advice.

[0843] Step 12:

[0844] The server sends the generated advice back to the terminal, which displays it to the user. The input is the advice data, and the output is the advice displayed in the user interface.

[0845] This process allows users to efficiently select ingredients in-store and receive real-time support for any cooking questions or difficulties after purchase.

[0846] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0847] System Overview

[0848] The system of the present invention, which aims to maximize the use of ingredients on hand and provide personalized recipes, integrates image recognition technology, generative AI models, real-time cooking support, and an emotion engine that recognizes user emotions.

[0849] System Components

[0850] 1. Users

[0851] Use a camera (such as a smartphone camera) to take a picture of the ingredients you own.

[0852] The captured image is sent to the server via the terminal.

[0853] Take a photo of any questions or problems that arise while cooking and enter them into the device as a question.

[0854] Enter any necessary preferences, allergies, and emotional information into the device.

[0855] The system provides cooking guidance based on skill level and relaxing recipes.

[0856] 2. Terminal

[0857] Images taken by the user are preprocessed and sent to the server.

[0858] Display received ingredient lists and customized recipes to users.

[0859] It receives questions and related images from users while they are cooking and sends them to the server.

[0860] Display advice and guidance received from the server.

[0861] Collects user emotion data and sends it to the server.

[0862] Providing cooking support based on the user's emotional state.

[0863] 3. Server

[0864] An image recognition algorithm is used to identify ingredients from the received image.

[0865] A customized recipe is generated based on the identified ingredient list.

[0866] Analyzes user questions and images to generate appropriate advice and solutions.

[0867] The generated ingredient list, customized recipe, and advice are sent back to the device.

[0868] The contents of the customized recipe are adjusted taking into account the user's preferences and allergy information.

[0869] It uses an emotion engine to analyze the user's emotional state and provides cooking support and recipes based on the results.

[0870] Program processing and specific examples

[0871] Image recognition and food ingredient identification

[0872] 1. The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo with their smartphone camera.

[0873] 2. The device preprocesses the photo (adjusts resolution, converts format, etc.) and sends it to the server.

[0874] 3. The server applies an image recognition algorithm (e.g., YOLO, ResNet, etc.) to analyze the submitted image and identify the ingredients in the photo. Let's say the identified ingredients are tomatoes, chicken, and carrots.

[0875] 4. The server generates a list of the identified ingredients and returns it to the terminal.

[0876] Customized recipe generation

[0877] 1. The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[0878] 2. The device sends the ingredient list and the user's preferences and allergy information to the server.

[0879] 3. The server uses a generative AI model (e.g., GPT-4 or Fine-Tuned model) based on the received information to generate a customized recipe. For example, a recipe for "Tomato and Chicken Stew" using tomatoes and chicken is created.

[0880] 4. The server returns the generated recipe (ingredients, steps, cooking time, etc.) to the terminal.

[0881] 5. The device displays the received recipe to the user.

[0882] Real-time cooking support

[0883] 1. When a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and enters the question into the device.

[0884] 2. The device sends the captured photo and question to the server.

[0885] 3. The server analyzes the received photo and question and generates appropriate advice, such as "Please continue baking a little longer."

[0886] 4. The server sends the generated advice to the terminal.

[0887] 5. The device displays the advice to the user.

[0888] Supported by an emotional engine

[0889] 1. The user inputs emotional information (e.g., feeling stressed or tired) into the device.

[0890] 2. The device sends the user's emotional information to the server.

[0891] 3. The server uses an emotion engine to analyze the user's emotional state and provide appropriate support based on that state. For example, a user feeling stressed might be suggested a recipe for a relaxing herbal tea.

[0892] 4. The server sends the generated cooking instructions and recipes back to the device.

[0893] 5. The device displays the received support and recipes to the user.

[0894] As described above, the system of the present invention combines image recognition technology, generative AI models, real-time cooking support, and an emotion engine to maximize the use of ingredients in the home and provide an efficient and enjoyable cooking experience, thereby reducing food waste and promoting healthy eating habits.

[0895] The processing flow will be explained below.

[0896] Image recognition and food ingredient identification

[0897] Step 1:

[0898] The user takes some ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[0899] Step 2:

[0900] The device pre-processes this photo (adjusting resolution, converting format, etc.) and sends it to the server.

[0901] Step 3:

[0902] The server analyzes the received image using an image recognition algorithm (e.g., YOLO, ResNet, etc.) and identifies the ingredients in the photo. Let's say the identified ingredients are tomatoes, chicken, and carrots.

[0903] Step 4:

[0904] The server generates a list of the identified ingredients and returns it to the terminal.

[0905] Process for generating customized recipes

[0906] Step 5:

[0907] The user checks the list of ingredients displayed on the device and enters their preferences and allergy information (e.g., I don't like spicy food).

[0908] Step 6:

[0909] The terminal sends a request to the server along with a list of ingredients based on the entered preferences and allergy information.

[0910] Step 7:

[0911] The server uses a generative AI model (e.g., GPT-4 or Fine-Tuned model) based on the received information to generate a customized recipe, such as a "Tomato and Chicken Stew" recipe using tomatoes and chicken.

[0912] Step 8:

[0913] The server returns the generated customized recipe (ingredients, steps, cooking time, etc.) to the terminal.

[0914] Step 9:

[0915] The device displays the received recipe to the user.

[0916] Real-time cooking support processing

[0917] Step 10:

[0918] If a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and types the question into the device.

[0919] Step 11:

[0920] The device sends the captured photo and the question to the server.

[0921] Step 12:

[0922] The server analyzes the received photos and questions and generates appropriate advice, such as "Please continue baking a little longer."

[0923] Step 13:

[0924] The server transmits the generated advice to the terminal.

[0925] Step 14:

[0926] The device displays the advice to the user.

[0927] Support processing with emotion engine

[0928] Step 15:

[0929] The user inputs emotional information (e.g., feeling stressed or tired) into the device.

[0930] Step 16:

[0931] The terminal transmits the user's emotion information to the server.

[0932] Step 17:

[0933] The server uses an emotion engine to analyze the user's emotional state and provide appropriate support based on that state. For example, a user feeling stressed might be suggested a recipe for a relaxing herbal tea.

[0934] Step 18:

[0935] The server returns the generated cooking instructions and recipes to the terminal.

[0936] Step 19:

[0937] The device displays the received support and recipes to the user.

[0938] In this way, the present invention performs a series of processes to maximize the use of ingredients available to the user and provide an efficient and enjoyable cooking experience. By combining image recognition technology, generative AI models, real-time cooking support, and an emotion engine, it is possible to provide appropriate support and recipe suggestions based on the user's emotional state. This reduces food waste and promotes healthy eating habits.

[0939] Example 2

[0940] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0941] Conventional cooking support systems make it difficult for users to efficiently utilize the individual ingredients they own, and also make it difficult to receive immediate and appropriate support for questions or difficulties that arise during cooking. Furthermore, they are unable to adequately respond to users' emotional state or individual preferences, and are therefore unable to increase satisfaction with cooking. This has led to problems such as wasted ingredients, stress during the cooking process, and a lack of recipes tailored to individual needs.

[0942] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0943] In this invention, the server includes a means for analyzing images using an image recognition algorithm to identify ingredients, a means for generating customized recipes based on the ingredients identified by the generative AI model, and a means for analyzing the user's emotional information using an emotion engine, thereby enabling efficient use of ingredients, real-time cooking support, and recipe provision according to the user's emotional state and preferences.

[0944] "User" refers to an individual who uses the system to receive support in managing ingredients and cooking.

[0945] "Photography device" refers to a device such as a camera or smartphone used to take an image of an ingredient.

[0946] "Image pre-processing" refers to processing such as adjusting the resolution and converting the format of a captured image.

[0947] "Image recognition algorithms" refers to machine learning models and techniques (e.g., YOLO, ResNet) used to identify objects in images.

[0948] "Ingredients" refers to food owned or purchased by the User that is used as an ingredient in cooking.

[0949] "Generative AI Model" refers to an artificial intelligence model (e.g., GPT-4) that generates customized recipes based on identified ingredients.

[0950] "Customized Recipe" refers to a personalized cooking guide generated based on identified ingredients and taking into account the user's preferences and allergy information.

[0951] "Emotion engine" refers to technology that analyzes the user's emotional information and provides appropriate cooking support and recipes based on that state.

[0952] "Real-time cooking support" refers to the function that provides immediate and appropriate advice and solutions to questions or problems that arise while cooking.

[0953] "Preference and Allergy Information" means information related to individual dietary preferences and allergies that a User enters into the System.

[0954] The system of the present invention is designed to maximize the use of ingredients available to users and provide personalized recipes. The system integrates image recognition technology, generative AI models, real-time cooking support functions, and an emotion engine that recognizes user emotions.

[0955] System configuration

[0956] user

[0957] Users take pictures of ingredients they own using a camera such as a smartphone and send them to the device. A smartphone camera is a common camera. If a user has a question or problem while cooking, they can take a picture of the relevant item and enter it as a question into the device. In addition, users can enter information about their preferences, allergies, and emotions into the device.

[0958] Terminal

[0959] The device preprocesses the images taken by the user and sends them to the server. Preprocessing includes adjusting the resolution and converting the format (e.g., from JPEG to PNG). The device displays the received ingredient list and customized recipe, and sends questions and related images from the user while cooking to the server. The device also displays advice and guidance received from the server. Furthermore, the device collects the user's emotional data and sends it to the server.

[0960] server

[0961] The server uses an image recognition algorithm (e.g., YOLO, ResNet) to analyze the image sent by the user and identify the ingredients. Based on the identified ingredient list, the server generates a customized recipe using a generative AI model (e.g., GPT-4). The generated recipe includes ingredients, steps, cooking time, etc. and is sent to the device. The server also analyzes the user's questions and images, generates appropriate advice, and sends it to the device. Furthermore, the server uses an emotion engine to analyze the user's emotional state and provides cooking support and recipes based on the results.

[0962] Specific examples

[0963] Image recognition and food ingredient identification

[0964] 1. The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo with their smartphone camera.

[0965] 2. The device preprocesses the captured photo (e.g., adjusts resolution, converts format) and sends it to the server.

[0966] 3. The server applies an image recognition algorithm to identify the ingredients in the photo. Let's say the ingredients are tomatoes, chicken, and carrots.

[0967] 4. The server generates a list of the identified ingredients and returns it to the terminal.

[0968] Customized recipe generation

[0969] 1. The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[0970] 2. The device sends the ingredient list and the user's preferences and allergy information to the server.

[0971] 3. The server uses the generative AI model based on the received information to generate a customized recipe, for example, a "Chicken and Tomato Stew" recipe.

[0972] 4. The server sends the generated recipe (ingredients, steps, cooking time, etc.) to the terminal.

[0973] 5. The device displays the received recipe to the user.

[0974] Real-time cooking support

[0975] 1. When a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and enters the question into the device.

[0976] 2. The device sends the captured photo and question to the server.

[0977] 3. The server analyzes the received photo and question and generates appropriate advice, such as "Please continue baking a little longer."

[0978] 4. The server sends the generated advice to the terminal.

[0979] 5. The device displays the advice to the user.

[0980] Supported by an emotional engine

[0981] 1. The user inputs emotional information (e.g., feeling stressed or tired) into the device.

[0982] 2. The device sends the user's emotional information to the server.

[0983] 3. The server uses an emotion engine to analyze the user's emotional state and provide appropriate support based on that state. For example, a user feeling stressed might be suggested a recipe for a relaxing herbal tea.

[0984] 4. The server sends the generated cooking instructions and recipes back to the device.

[0985] 5. The device displays the received support and recipes to the user.

[0986] Prompt Sentence Examples

[0987] 1. Image recognition prompt:

[0988] "Please identify the ingredients in this image"

[0989] 2. Recipe generation prompt:

[0990] "Please suggest some non-spicy recipes using these ingredients."

[0991] 3. Cooking support prompt:

[0992] "Please judge whether the chicken in this photo is cooked thoroughly."

[0993] 4. Emotion Engine Support Prompt:

[0994] "Please suggest recipes that will help users relax when they are feeling stressed."

[0995] By using this system, users can use the ingredients they have on hand without waste and enjoy healthy and delicious meals.

[0996] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0997] Step 1:

[0998] The user takes the ingredients they plan to use (e.g., tomatoes, chicken, carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[0999] Input: Images of ingredients in the refrigerator

[1000] Output: Photographed food image (JPEG format)

[1001] Specific actions: Open the camera app on your smartphone, arrange the ingredients, take a photo, and press the shutter button with a "click!"

[1002] Step 2:

[1003] The device pre-processes the captured image, which includes adjusting the resolution and converting the format from JPEG to PNG.

[1004] Input: Photographed food image (JPEG format)

[1005] Output: Preprocessed food image (PNG format)

[1006] Specific operation: Run a program to adjust the resolution and convert the file format to PNG. The resolution adjustment is, for example, to make the width and height 800 pixels each.

[1007] Step 3:

[1008] The terminal sends the preprocessed image to the server and generates a prompt to start the image recognition algorithm.

[1009] Input: Preprocessed food image (PNG format)

[1010] Output: Image data and prompt text are sent to the server

[1011] Specific operation: The preprocessed image is sent to the server via an HTTP request, and a prompt message is sent saying, "Please identify the ingredients from this image."

[1012] Step 4:

[1013] The server applies an image recognition algorithm (e.g., YOLO, ResNet) to analyze the received image.

[1014] Input: Preprocessed food image, prompt

[1015] Output: List of recognized ingredients (e.g., tomato, chicken, carrot)

[1016] What it does: Runs the YOLO algorithm to detect objects in an image and generate an ingredients list. YOLO identifies tomatoes, chicken, and carrots in the image.

[1017] Step 5:

[1018] The server generates a customized recipe using a generative AI model (e.g., GPT-4) based on the identified ingredient list.

[1019] Input: List of recognized ingredients (e.g., tomato, chicken, carrot)

[1020] Output: Customized recipe (ingredients, steps, cooking time, etc.)

[1021] Specific behavior: GPT-4 is fed a list of ingredients and generates a recipe using the prompt "Please suggest a non-spicy recipe using these ingredients." The recipe "Tomato and Chicken Stew" is generated.

[1022] Step 6:

[1023] The server transmits the generated recipe to the terminal.

[1024] Input: Generated custom recipe

[1025] Output: Customized recipe data is sent to the device

[1026] What it does: It uses HTTP push notifications to send the generated recipe information to the device, including the ingredients list, instructions, cooking time, etc.

[1027] Step 7:

[1028] The device displays the received recipe to the user.

[1029] Input: Customized recipe data

[1030] Output: The recipe is displayed on the smartphone screen.

[1031] Specific Behavior: Runs a screen display program to format and display the received recipe information to the user. The UI includes the ingredient list, instructions, and cooking time.

[1032] Step 8:

[1033] If a user has a question while cooking, they can enter it, take a picture of the relevant information, and send it to the device.

[1034] Input: Cooking question or related image

[1035] Output: Questions and image data are sent from the device to the server.

[1036] How it works: You enter your question into your smartphone and take a photo of the related cooking state. The device then sends these to the server.

[1037] Step 9:

[1038] The server analyzes the received question and image and generates appropriate advice.

[1039] Input: Question and associated image data

[1040] Output: Advice (e.g. "Please continue to cook a little longer")

[1041] Specific operation: Analyzes the image using an image analysis algorithm and generates advice based on the question. Appropriate cooking advice is generated in text format.

[1042] Step 10:

[1043] The server transmits the generated advice to the terminal.

[1044] Input: Advice content

[1045] Output: Advice data is sent to the terminal

[1046] Specific operation: Advice information is sent to the device using HTTP push notifications.

[1047] Step 11:

[1048] The device displays the received advice to the user.

[1049] Input: Advice data

[1050] Output: Advice content is displayed on the smartphone screen

[1051] Specific behavior: Run the screen display program, format the received advice information, and display it to the user. The UI includes the advice text.

[1052] Step 12:

[1053] The user inputs their emotional state (e.g., feeling stressed) into the terminal.

[1054] Input: Emotion information

[1055] Output: Emotional information data is sent from the device to the server.

[1056] Specific actions: Enter your emotional state using the multiple choice input form and press the submit button.

[1057] Step 13:

[1058] The server analyzes the emotional information and generates cooking support and relaxing recipes according to the state of the user.

[1059] Input: Emotional information data

[1060] Output: Cooking support and relaxing recipes

[1061] Specific behavior: Activates the emotion engine to analyze the user's emotional state. Based on the analysis results, it generates a relaxing recipe (e.g., "How to make herbal tea").

[1062] Step 14:

[1063] The server transmits the generated cooking instructions and recipes to the terminal.

[1064] Input: Cooking support and relaxing recipes

[1065] Output: Support and recipe data sent to device

[1066] Specific operation: Send cooking support information to the device using HTTP push notifications.

[1067] Step 15:

[1068] The device displays the received support and recipes to the user.

[1069] Input: Cooking support and relaxation recipe data

[1070] Output: Support and recipe information displayed on the smartphone screen

[1071] Specific behavior: Executes the screen display program, formats the received support and recipe information, and displays it to the user. The UI includes support text and relaxation recipe text.

[1072] Through these processing steps, users can receive personalized recipes and real-time cooking support while making the most of the ingredients they have on hand. The system provides efficient ingredient utilization and a comfortable cooking experience.

[1073] (Application example 2)

[1074] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1075] Conventional cooking assistance systems are limited to providing recipes based on the ingredients a user has on hand, and are unable to provide real-time support during cooking or cooking assistance that takes into account the user's emotional state. Furthermore, they are unable to generate recipes that fully reflect the user's preferences and allergies. This has led to problems such as increased stress due to the difficulty users experience while cooking and a lack of suitable recipes.

[1076] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to take images of ingredients owned by the user using a camera, means for receiving the taken images and identifying the ingredients using image recognition technology, means for providing the user with a customized recipe generated based on the identified ingredients, means for generating a customized recipe based on the ingredient list and user information using a generative AI model, and means for inputting the user's emotional information and automatically adjusting the cooking support content provided using an emotion analysis engine. This enables more appropriate and personalized recipes and real-time cooking support that reflect the user's individual needs and preferences, as well as cooking support that responds to the user's emotional state.

[1077] A "photography device" is a device used to take pictures of ingredients owned by a user, such as a smartphone or camera.

[1078] "Image recognition technology" refers to algorithms and techniques for identifying and classifying specific objects or ingredients from photographed images, and includes models such as YOLO and ResNet.

[1079] A "generative AI model" is an artificial intelligence model that generates new text or recipes based on user input or identified data, and examples include GPT-4.

[1080] An "emotion analysis engine" is a technology that analyzes the emotional information entered by the user and provides appropriate cooking support and recipe adjustments based on the results.

[1081] A "customized recipe" is a recipe that meets a user's individual needs and is generated based on identified ingredients, the user's preferences, allergy information, and other individual information.

[1082] "Real-time cooking support" is a service that provides instant solutions and advice to users for any questions or difficulties they may have while cooking.

[1083] "User Information" means information about an individual user, such as preferences, allergy information, and emotional state, that is necessary to provide customized recipes and cooking support.

[1084] The "ingredient list" is a list of ingredients owned by the user, identified using image recognition technology.

[1085] The system embodying this invention integrates image recognition technology, generative AI models, real-time cooking support, and a sentiment analysis engine to recognize user emotions, in order to maximize the use of ingredients on hand and provide personalized recipes.

[1086] System Components

[1087] 1. Users

[1088] Use a camera (such as a smartphone camera) to take a picture of the ingredients you own.

[1089] The captured image is sent to the server via the terminal.

[1090] Take a photo of any questions or problems that arise while cooking and enter them into the device as a question.

[1091] Emotional information, such as stress level or relaxation level, is also entered into the device.

[1092] 2. Terminal

[1093] Images taken by the user are preprocessed and sent to the server.

[1094] Display received ingredient lists and customized recipes to users.

[1095] Collects user emotion data and sends it to the server.

[1096] 3. Server

[1097] Using an image recognition algorithm (e.g., YOLO, ResNet), ingredients are identified from the received image.

[1098] Generate a customized recipe using a generative AI model (e.g., GPT-4) based on the identified list of ingredients.

[1099] An emotion analysis engine that analyzes the user's emotional state is used to adjust the generated recipes and cooking support content.

[1100] Hardware and software used

[1101] Camera: Smartphone camera

[1102] Device: Smartphone or tablet

[1103] Server-side software:

[1104] Image recognition algorithms (e.g., YOLO, ResNet)

[1105] Generative AI models (e.g., GPT-4)

[1106] Sentiment Analysis Engine

[1107] Specific examples

[1108] 1. Image Recognition and Food Identification

[1109] The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[1110] The device preprocesses the photo (adjusting resolution, converting format, etc.) and sends it to the server.

[1111] The server applies an image recognition algorithm to identify the ingredients in the photo, which may be tomatoes, chicken, and carrots.

[1112] 2. Customized recipe generation

[1113] The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[1114] The device sends the ingredient list and the user's preferences and allergy information to the server.

[1115] The server uses the generative AI model to generate customized recipes, such as a "Tomato and Chicken Stew" recipe using tomatoes and chicken.

[1116] An example of a prompt sentence that will generate a recipe is:

[1117] Ingredients: Tomato, Chicken, Carrot

[1118] User Info: {"preferences": ["I hate spicy food"], "allergies": ["nuts"], "emotion_text": "I'm stressed"}

[1119] Recipe:

[1120] 3. Real-time cooking support

[1121] If a user has a question while cooking (e.g., "Is this chicken cooked enough?"), they take a photo of the cooked chicken and type the question into the device.

[1122] The device sends the captured photo and question to the server.

[1123] The server analyzes the received photo and question and generates appropriate advice, such as "Please continue baking a little longer."

[1124] 4. Support with sentiment analysis

[1125] The user's emotional information (e.g., feeling stressed) is input into the device.

[1126] The device transmits the user's emotional information to the server.

[1127] The server uses an emotion analysis engine to analyze the user's emotional state and provides cooking support and recipes based on the results.

[1128] The system allows users to make the most of the ingredients they have and easily obtain personalized recipes. It also responds to questions while cooking in real time and provides cooking support that adapts to their emotional state, making the cooking experience more efficient and enjoyable while reducing food waste.

[1129] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1130] Step 1:

[1131] Users take ingredients from the refrigerator and take a picture of them with their smartphone camera, which then stores the image on the device.

[1132] Step 2:

[1133] The device pre-processes the captured image, adjusts the resolution, and converts it to the appropriate format, then sends the pre-processed image to the server.

[1134] Step 3:

[1135] The server analyzes the received image using an image recognition algorithm (e.g., YOLO, ResNet), and generates a list of ingredients identified from the image and sends it back to the device.

[1136] Step 4:

[1137] Users can review the list of identified ingredients and enter their preferences and allergies into the device, including details such as an aversion to spicy food.

[1138] Step 5:

[1139] The device sends the input user information (ingredients list, preferences, allergy information) to the server, which uses a generative AI model (e.g., GPT-4) to generate a customized recipe based on the input information.

[1140] Step 6:

[1141] The server sends the created customized recipe to the terminal, which displays it to the user and prompts for confirmation.

[1142] Step 7:

[1143] When a user has a question while cooking, they can take a picture of the relevant food and enter it into the device along with their question.

[1144] Step 8:

[1145] The device sends the captured image and question to the server, which analyzes the received data, generates appropriate solutions or advice, and sends it back to the device.

[1146] Step 9:

[1147] The user inputs emotional information (e.g., feeling stressed) into the device, which then transmits this information to the server.

[1148] Step 10:

[1149] The server uses an emotion analysis engine to analyze the user's emotional state, and based on the results, adjusts cooking support and recipe content and sends it back to the device.

[1150] Step 11:

[1151] The device displays the adjusted recipe and cooking support information to the user, who then proceeds with cooking based on this information.

[1152] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1153] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1154] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1155] [Third embodiment]

[1156] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1157] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1158] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1159] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1160] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1161] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1162] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1163] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1164] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1165] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1166] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1167] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1168] System Overview

[1169] The "SmartChef" system of the present invention is a system that maximizes the use of ingredients on hand and provides personalized recipes. The system integrates image recognition technology, generative AI models, and real-time cooking support functions.

[1170] System Components

[1171] 1. Users

[1172] Use a camera (such as a smartphone camera) to take a picture of the ingredients you own.

[1173] The captured image is sent to the server via the terminal.

[1174] Take a photo of any questions or problems that arise while cooking and enter them into the device as a question.

[1175] Enter any necessary preferences or allergy information into the device.

[1176] 2. Terminal

[1177] Images taken by the user are preprocessed and sent to the server.

[1178] Display received ingredient lists and customized recipes to users.

[1179] It receives questions and related images from users while they are cooking and sends them to the server.

[1180] Display advice and guidance received from the server.

[1181] 3. Server

[1182] An image recognition algorithm is used to identify ingredients from the received image.

[1183] A customized recipe is generated based on the identified ingredient list.

[1184] Analyzes user questions and images to generate appropriate advice and solutions.

[1185] The generated ingredient list, customized recipe, and advice are sent back to the device.

[1186] The contents of the customized recipe are adjusted taking into account the user's preferences and allergy information.

[1187] Specific processing flow

[1188] Image recognition and food ingredient identification

[1189] 1. The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo with their smartphone camera.

[1190] 2. The device preprocesses the photo (adjusts resolution, converts format, etc.) and sends it to the server.

[1191] 3. The server receives the submitted image and applies image recognition algorithms to identify the ingredients in the photo, such as tomatoes, chicken, and carrots.

[1192] 4. The server generates a list of the identified ingredients and returns it to the terminal.

[1193] Customized recipe generation

[1194] 1. The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[1195] 2. The terminal sends the ingredient list and the user's preferences and allergy information to the server.

[1196] 3. The server uses the generative AI model based on the received information to generate a customized recipe, such as a "Tomato and Chicken Stew" recipe.

[1197] 4. The server returns the generated recipe (ingredients, steps, cooking time, etc.) to the terminal.

[1198] 5. The device displays the received recipe to the user.

[1199] Real-time cooking support

[1200] 1. When a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and enters the question into the device.

[1201] 2. The device sends the captured photo and question to the server.

[1202] 3. The server analyzes the received photo and question and generates appropriate advice. For example, an AI algorithm determines the doneness of the chicken and generates advice such as "Continue cooking a little longer."

[1203] 4. The server sends the generated advice to the terminal.

[1204] 5. The device displays the advice to the user.

[1205] Educational support and cooking skill development

[1206] 1. The server analyzes data about the user's past cooking history and current skill level.

[1207] 2. The server generates cooking guidance based on the user's current skill level (e.g., how to make a sauce or adjust the heat).

[1208] 3. The server sends the generated guidance to the terminal.

[1209] 4. The device displays the received guidance to the user and provides assistance in improving cooking skills step by step.

[1210] As described above, the "SmartChef" system of the present invention combines image recognition technology, generative AI models, and real-time cooking support to maximize the use of ingredients users have at home and provide an efficient and enjoyable cooking experience, thereby reducing food waste and promoting healthy eating habits.

[1211] The processing flow will be explained below.

[1212] Step 1:

[1213] The user takes some ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[1214] Step 2:

[1215] The device preprocesses the captured photos (adjusting the resolution, converting the format, etc.) and sends them to the server.

[1216] Step 3:

[1217] The server applies an image recognition algorithm (e.g., YOLO, ResNet, etc.) to analyze the received image and identify the ingredients in the photo. Let's say the identified ingredients are tomatoes, chicken, and carrots.

[1218] Step 4:

[1219] The server generates a list of the identified ingredients and returns it to the terminal.

[1220] Step 5:

[1221] The user checks the list of ingredients displayed on the device and enters their preferences and allergy information (e.g., I don't like spicy food).

[1222] Step 6:

[1223] The terminal transmits the ingredient list and the user's preferences and allergy information to the server.

[1224] Step 7:

[1225] The server uses a generative AI model (e.g., GPT-4 or Fine-Tuned model) based on the received information to generate a customized recipe, such as a "Tomato and Chicken Stew" recipe.

[1226] Step 8:

[1227] The server sends the generated recipe (ingredients, steps, cooking time, etc.) back to the terminal.

[1228] Step 9:

[1229] The device displays the received recipe to the user.

[1230] Step 10:

[1231] If a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and types the question into the device.

[1232] Step 11:

[1233] The device sends the captured photo and the question to the server.

[1234] Step 12:

[1235] The server analyzes the received photos and questions and generates appropriate advice, such as "Please continue baking a little longer."

[1236] Step 13:

[1237] The server transmits the generated advice to the terminal.

[1238] Step 14:

[1239] The device displays the advice to the user.

[1240] Step 15:

[1241] The server analyzes data about the user's past cooking history and current skill level.

[1242] Step 16:

[1243] The server generates cooking guidance based on the user's current skill level (e.g., how to make a sauce or adjust the heat).

[1244] Step 17:

[1245] The server transmits the generated guidance to the terminal.

[1246] Step 18:

[1247] The device displays the received guidance to the user, providing assistance in improving cooking skills step by step.

[1248] In this way, the invention combines image recognition technology, generative AI models, and real-time cooking support to maximize the use of ingredients a user has on hand and provide an efficient and enjoyable cooking experience, reducing food waste and promoting healthy eating habits.

[1249] Example 1

[1250] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1251] In modern households, it is often difficult to effectively utilize ingredients available in the refrigerator to prepare delicious and healthy meals. In particular, there is a need for systems that can suggest recipes based on ingredients, resolve cooking questions, and accommodate individual user preferences and allergies. However, conventional systems lack systems that meet these requirements, resulting in increased food waste and cooking stress for users. The present invention aims to solve the above problems and provide users with a convenient and effective cooking assistance system.

[1252] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1253] In this invention, the server includes a means for a user to take images of ingredients owned by the user using a camera, a means for receiving the taken images and identifying the ingredients using image recognition technology, and a means for generating customized recipes using a generative AI model based on the identified ingredients and the user's preferences and allergy information, and providing the recipes to the user, thereby enabling the user to optimally utilize ingredients on hand and receive personalized recipes and cooking support.

[1254] "User" refers to the end user who uses the system to manage ingredients and cook.

[1255] A "photography device" is a device owned by a user for taking pictures of ingredients, and includes, for example, a smartphone camera.

[1256] "Image recognition technology" refers to algorithms and methods for automatically analyzing and identifying ingredients contained in a photographed image.

[1257] "Generative AI Model" refers to an artificial intelligence model used to generate customized recipes based on identified ingredients and user information.

[1258] A "customized recipe" is a recipe that includes individual cooking instructions and an ingredient list that are generated based on the ingredients a user has on hand and that takes into account the user's preferences and allergy information.

[1259] "Cooking Support" is a function that provides appropriate advice and solutions in real time to users when they have questions or difficulties while cooking.

[1260] "Guidance" refers to specific instruction and advice provided to improve a user's cooking skills.

[1261] "Ingredients" are food ingredients that are photographed by the user using a camera and recognized by the system.

[1262] "Preferences and allergy information" refers to information about a user's individual dietary preferences and allergies that the user enters in advance.

[1263] The system of the present invention is designed to maximize the use of ingredients on hand by the user and provide personalized recipes. This system consists of three main components: the user, the terminal, and the server, and their cooperation provides advanced cooking support.

[1264] Overview of the hardware and software used

[1265] 1. User:

[1266] Photography device: Use a device such as a smartphone camera to take pictures of ingredients you own.

[1267] Input device: Use a smartphone or tablet touch screen, keyboard, etc. to input preferences, allergy information, and questions during cooking.

[1268] 2. Device (smartphone, tablet, PC, etc.):

[1269] Image pre-processing software: Adjusts the resolution of received images, converts formats, and removes noise.

[1270] User interface: Displays ingredient lists, customizable recipes, and real-time advice.

[1271] 3. Server:

[1272] Image recognition algorithm: Using libraries such as TensorFlow, ingredients are identified from the received images.

[1273] Generative AI model: Utilizing GPT-3 and other models, it generates customized recipes based on identified ingredients, user preferences, and allergy information.

[1274] Data analysis unit: Analyzes the user's past cooking history and skill level and provides educational guidance.

[1275] Specific processing flow

[1276] Image recognition and food ingredient identification

[1277] 1. User:

[1278] Take some ingredients (e.g. tomatoes, chicken, carrots) out of the refrigerator and take a photo with your smartphone camera.

[1279] 2. Terminal:

[1280] Preprocess the photos (adjust resolution, convert format, etc.) and send them to the server.

[1281] 3. Server:

[1282] The received image is then processed using image recognition algorithms such as TensorFlow to identify ingredients, such as tomatoes, chicken, and carrots.

[1283] The identified ingredient list is returned to the terminal.

[1284] Customized recipe generation

[1285] 1. User:

[1286] Check the list of identified ingredients and enter your preferences and allergy information into the device (e.g., I don't like spicy food).

[1287] 2. Terminal:

[1288] A list of ingredients and information on preferences and allergies is sent to the server.

[1289] 3. Server:

[1290] A generative AI model (e.g., GPT-3) is used to generate a customized recipe, for example, "Chicken and Tomato Stew."

[1291] The generated recipe (ingredients, steps, cooking time, etc.) is sent back to the device.

[1292] 4. Terminal:

[1293] Display the received recipe to the user.

[1294] Real-time cooking support

[1295] 1. User:

[1296] If you have any questions while cooking (e.g., "Is this chicken cooked enough?"), take a photo of the cooking process, enter your question, and send it to the device.

[1297] 2. Terminal:

[1298] Send the question and image to the server.

[1299] 3. Server:

[1300] It uses AI algorithms to analyze the question and generate appropriate advice (e.g., "Please continue cooking a little longer").

[1301] The generated advice is sent to the terminal.

[1302] 4. Terminal:

[1303] Display advice to the user.

[1304] Educational support and cooking skill development

[1305] 1. Server:

[1306] Analyze data about your cooking history and current skill level.

[1307] Generate cooking guidance based on skill level (e.g., how to make a sauce or adjust the heat).

[1308] Send guidance to the device.

[1309] 2. Terminal:

[1310] The received guidance is displayed to the user, providing assistance in improving cooking skills step by step.

[1311] Specific examples

[1312] For example, if a user takes an image of a tomato, chicken, and carrot, the following prompt sentences are input into the generative AI model:

[1313] Example prompt sentence:

[1314] "Generate a non-spicy recipe using tomatoes, chicken, and carrots."

[1315] This prompt will cause the generative AI model to generate a recipe for "Tomato and Chicken Stew" and send it to the device.

[1316] As described above, by providing advanced cooking support through collaboration between the user, device, and server, users can make optimal use of the ingredients they have on hand and receive personalized recipes and cooking support.

[1317] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1318] Step 1:

[1319] The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a picture of the ingredients with the smartphone camera. The user then sends the image to the device via the "SmartChef" app. The input data is the image of the ingredients, and the output data is the image file sent to the device.

[1320] Step 2:

[1321] The terminal preprocesses the received image. Preprocessing includes adjusting the image resolution, format conversion, and noise removal, making the image suitable for analysis. The input data is the image file sent by the user, and the output data is the preprocessed image file.

[1322] Step 3:

[1323] The device sends the preprocessed image file to the server, which uses an image recognition algorithm such as TensorFlow to identify ingredients from the received image. The input data is the preprocessed image file, and the output data is a list of identified ingredients.

[1324] Step 4:

[1325] The server sends the identified ingredient list to the terminal. The terminal displays the ingredient list to the user. The user checks the displayed ingredient list, checks for missing information or errors, and corrects it as necessary. The input data is the ingredient list sent from the server, and the output data is the corrected ingredient list.

[1326] Step 5:

[1327] The user inputs their preferences and allergy information into the terminal. For example, they input "I don't like spicy food." The terminal then sends the input preference and allergy information along with the ingredient list to the server. The input data is the user's preference and allergy information and the ingredient list, and the output data is the information sent to the server.

[1328] Step 6:

[1329] The server generates a customized recipe using a generative AI model (e.g., GPT-3) based on the received information. The input data is the user's preferences, allergy information, and a list of ingredients, and the output data is the generated customized recipe. For example, a recipe called "Tomato and Chicken Stew" is generated.

[1330] Step 7:

[1331] The server sends the generated customized recipe to the terminal. The terminal displays the received recipe to the user. The input data is the generated customized recipe, and the output data is the recipe displayed on the terminal.

[1332] Step 8:

[1333] When a question arises during cooking (e.g., "Is this chicken cooked thoroughly?"), the user takes a photo of the chicken being cooked and enters the question into the terminal and sends it. The input data is the image of the chicken being cooked and the question, and the output data is the data sent to the terminal.

[1334] Step 9:

[1335] The device sends the captured photo and question to the server, which then uses an AI algorithm to analyze the image and question and generate appropriate advice. For example, the advice generated might be, "Please continue cooking a little longer." The input data is the image of the chicken and the question, and the output data is the generated advice.

[1336] Step 10:

[1337] The server sends the generated advice to the terminal. The terminal displays the received advice to the user. The input data is the generated advice, and the output data is the advice displayed on the terminal.

[1338] Step 11:

[1339] The server analyzes data related to the user's cooking history and current skill level and generates guidance for skill improvement. For example, guidance on "how to make a sauce and how to adjust the heat" is generated. The input data is the user's cooking history and skill level, and the output data is the generated guidance.

[1340] Step 12:

[1341] The server transmits the generated guidance to the terminal. The terminal displays the received guidance to the user, providing support for step-by-step improvement of cooking skills. The input data is the generated guidance, and the output data is the guidance displayed on the terminal.

[1342] (Application example 1)

[1343] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1344] In today's busy lifestyles, consumers are expected to prepare healthy, balanced meals while using their time efficiently. However, many difficulties and questions often arise during the process from purchasing ingredients to cooking. In particular, users feel anxious about selecting ingredients, and there is a lack of support that can address questions they have while cooking in real time. This can lead to food waste and make it difficult to prepare healthy meals. To solve these problems, the present invention aims to provide a system that supports the entire process from ingredient selection to cooking in a physical store, allowing users to select ingredients with confidence and cook effectively.

[1345] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1346] In this invention, the server includes means for a user to take images of ingredients owned by the user using a camera, means for receiving the taken images and identifying the ingredients using image recognition technology, means for providing the user with a customized recipe generated based on the identified ingredients, means for inputting the user's preferences and allergy information, means for adjusting the generation of the customized recipe taking into account the input preferences and allergy information, means for scanning ingredients in a physical store and providing an ingredient list and recipe on the spot, and means for resolving questions the user has about ingredient selection in real time. This allows the user to efficiently select ingredients in a physical store and receive real-time support for any questions or difficulties they may have about cooking even after purchase.

[1347] A "user" is a consumer who uses the system, inputs information related to ingredient selection and cooking, and receives feedback.

[1348] A "photography device" is a device owned by the user that includes a camera for taking pictures of ingredients and cooking conditions, and includes smartphones and tablets.

[1349] "Image recognition technology" is a technology that uses computer vision and machine learning algorithms to identify specific ingredients from photographed images.

[1350] A "customized recipe" is a personalized recipe created based on the user's ingredients, preferences, and allergy information.

[1351] "Preference and allergy information" refers to information about the user's food preferences and allergies, and is taken into consideration when generating customized recipes.

[1352] A "physical store" is a physical store where users directly purchase ingredients, such as a supermarket or grocery store.

[1353] "Scanning" is the process by which a user takes a picture of an ingredient in a physical store using a camera, allowing the system to identify the ingredient.

[1354] "Real-time support" is a feature that provides immediate advice and solutions to questions or difficulties users may encounter while selecting ingredients or cooking.

[1355] The "SmartShop" system supports the workflow from ingredient selection to cooking in a brick-and-mortar store. The system consists of a user-specific application (hereafter referred to as a terminal) and a server that performs image recognition and recipe generation.

[1356] System configuration

[1357] The components of this system are as follows:

[1358] 1. Terminal

[1359] Camera: Uses a smartphone or tablet camera to capture the ingredients the user selects in the store and the cooking process.

[1360] User Interface: Displays the ingredient list and customized recipe, and provides a means for users to input preferences and allergy information.

[1361] Communication function: Sends captured images and questions to a server and receives responses from the server.

[1362] 2. Server

[1363] Image Recognition: Uses computer vision and machine learning algorithms to analyze user-submitted images and identify ingredients.

[1364] Generative AI model: Generates customized recipes based on identified ingredients and user preferences and allergies.

[1365] Real-time support function: Generates and provides immediate advice and solutions to any questions or difficulties encountered while cooking.

[1366] Program processing explanation

[1367] Hardware and Software

[1368] Smartphone / Tablet (e.g. iPhone, Samsung Galaxy, etc.): Serves as a user interface and capture device.

[1369] Server: Use a cloud-based server such as Amazon Web Services (AWS) or Google Cloud Platform (GCP).

[1370] Python + Flask: Used to build the backend API.

[1371] PIL (Python Imaging Library) and numpy: Used for image and data processing.

[1372] Computer vision technology: OpenCV and TensorFlow are used as algorithms for food ingredient recognition.

[1373] Generative AI models: Use generative AI, such as OpenAI's GPT-3, to generate customized recipes.

[1374] Data processing and calculation

[1375] The device takes pictures of ingredients and cooking conditions using the smartphone camera and sends the image data to the server, where it is pre-processed by adjusting the resolution and converting the format.

[1376] The server analyzes the received image using computer vision technology to identify the ingredients, using neural network models for ingredient identification (e.g., ResNet, YOLO, etc.).

[1377] The generative AI model takes the identified ingredient list and the user's preferences and allergy information as inputs to generate customized recipes, providing recipes tailored to the user's needs.

[1378] The real-time support function analyzes captured images and questions that users may have while cooking, and generates appropriate advice using a generative AI model, which is then sent back to the device.

[1379] Specific examples

[1380] Example 1: Use in a physical store

[1381] A user scans "tomatoes, chicken, and carrots" in a physical store using their smartphone. Based on this information and the user's pre-entered preferences (such as not liking spicy food), the system provides a recipe for "tomato and chicken stew." If the user has questions about how to check the freshness of tomatoes at the time of purchase, the server immediately returns advice.

[1382] Example prompt:

[1383] Ingredients: Tomato, chicken, carrot. Preferences: Don't like spicy food. Allergies: None. Please recommend some recipes using these ingredients.

[1384] This allows users to smoothly select ingredients and cook them, reducing food waste and providing an efficient and enjoyable cooking experience.

[1385] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1386] Step 1:

[1387] The user takes a photo of the ingredients they plan to purchase in a physical store using the smartphone camera. The user then takes a picture of the ingredients and saves the image on the device through the application. The input is the photographed image of the ingredients, and the output is preprocessed image data.

[1388] Step 2:

[1389] The device preprocesses the captured images and sends them to the server. Preprocessing includes image resolution adjustment, format conversion, noise removal, etc. The input is the captured raw image data, and the output is the preprocessed image data.

[1390] Step 3:

[1391] The server analyzes the received image data and identifies the ingredients using image recognition technology. Specifically, it uses computer vision algorithms (e.g., YOLO, ResNet) to identify the type of ingredients. The input is the preprocessed image data, and the output is a list of identified ingredients.

[1392] Step 4:

[1393] The server generates a list of identified ingredients and returns it to the device. The user can then check the list on the device. The input is the ingredient information identified by image recognition, and the output is the ingredient list.

[1394] Step 5:

[1395] The user inputs their preferences and allergy information into the terminal. The input is the user's preferences and allergies, and the output is the updated information.

[1396] Step 6:

[1397] The terminal transmits the ingredient list and the user's preferences and allergy information to the server. The input is the user input information and the specified ingredient list, and the output is the data sent to the server.

[1398] Step 7:

[1399] The server generates a customized recipe using a generative AI model based on the received information. The AI ​​model uses OpenAI's GPT-3 and other models to automatically generate recipes that take into account the identified ingredients and the user's preferences and allergy information. The input is the identified ingredient list and the user's preference data, and the output is a customized recipe.

[1400] Step 8:

[1401] The server returns the generated customized recipe to the terminal, which then displays the received recipe to the user. The input is the generated recipe information, and the output is the recipe displayed on the user interface.

[1402] Step 9:

[1403] If a user has a question or difficulty while cooking, they can take a photo and input the question into the device. The input is the cooking question and related images, and the output is the question data stored on the device.

[1404] Step 10:

[1405] The terminal sends the user's question and related images to the server. The input is the question data from the user and the related images, and the output is the data sent to the server.

[1406] Step 11:

[1407] The server analyzes the received questions and images and generates appropriate advice and solutions. It again uses a generative AI model to derive a specific solution to the question or problem. The input is the user's question data and related images, and the output is the generated advice.

[1408] Step 12:

[1409] The server sends the generated advice back to the terminal, which displays it to the user. The input is the advice data, and the output is the advice displayed in the user interface.

[1410] This process allows users to efficiently select ingredients in-store and receive real-time support for any cooking questions or difficulties after purchase.

[1411] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1412] System Overview

[1413] The system of the present invention, which aims to maximize the use of ingredients on hand and provide personalized recipes, integrates image recognition technology, generative AI models, real-time cooking support, and an emotion engine that recognizes user emotions.

[1414] System Components

[1415] 1. Users

[1416] Use a camera (such as a smartphone camera) to take a picture of the ingredients you own.

[1417] The captured image is sent to the server via the terminal.

[1418] Take a photo of any questions or problems that arise while cooking and enter them into the device as a question.

[1419] Enter any necessary preferences, allergies, and emotional information into the device.

[1420] The system provides cooking guidance based on skill level and relaxing recipes.

[1421] 2. Terminal

[1422] Images taken by the user are preprocessed and sent to the server.

[1423] Display received ingredient lists and customized recipes to users.

[1424] It receives questions and related images from users while they are cooking and sends them to the server.

[1425] Display advice and guidance received from the server.

[1426] Collects user emotion data and sends it to the server.

[1427] Providing cooking support based on the user's emotional state.

[1428] 3. Server

[1429] An image recognition algorithm is used to identify ingredients from the received image.

[1430] A customized recipe is generated based on the identified ingredient list.

[1431] Analyzes user questions and images to generate appropriate advice and solutions.

[1432] The generated ingredient list, customized recipe, and advice are sent back to the device.

[1433] The contents of the customized recipe are adjusted taking into account the user's preferences and allergy information.

[1434] It uses an emotion engine to analyze the user's emotional state and provides cooking support and recipes based on the results.

[1435] Program processing and specific examples

[1436] Image recognition and food ingredient identification

[1437] 1. The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo with their smartphone camera.

[1438] 2. The device preprocesses the photo (adjusts resolution, converts format, etc.) and sends it to the server.

[1439] 3. The server applies an image recognition algorithm (e.g., YOLO, ResNet, etc.) to analyze the submitted image and identify the ingredients in the photo. Let's say the identified ingredients are tomatoes, chicken, and carrots.

[1440] 4. The server generates a list of the identified ingredients and returns it to the terminal.

[1441] Customized recipe generation

[1442] 1. The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[1443] 2. The device sends the ingredient list and the user's preferences and allergy information to the server.

[1444] 3. The server uses a generative AI model (e.g., GPT-4 or Fine-Tuned model) based on the received information to generate a customized recipe. For example, a recipe for "Tomato and Chicken Stew" using tomatoes and chicken is created.

[1445] 4. The server returns the generated recipe (ingredients, steps, cooking time, etc.) to the terminal.

[1446] 5. The device displays the received recipe to the user.

[1447] Real-time cooking support

[1448] 1. When a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and enters the question into the device.

[1449] 2. The device sends the captured photo and question to the server.

[1450] 3. The server analyzes the received photo and question and generates appropriate advice, such as "Please continue baking a little longer."

[1451] 4. The server sends the generated advice to the terminal.

[1452] 5. The device displays the advice to the user.

[1453] Supported by an emotional engine

[1454] 1. The user inputs emotional information (e.g., feeling stressed or tired) into the device.

[1455] 2. The device sends the user's emotional information to the server.

[1456] 3. The server uses an emotion engine to analyze the user's emotional state and provide appropriate support based on that state. For example, a user feeling stressed might be suggested a recipe for a relaxing herbal tea.

[1457] 4. The server sends the generated cooking instructions and recipes back to the device.

[1458] 5. The device displays the received support and recipes to the user.

[1459] As described above, the system of the present invention combines image recognition technology, generative AI models, real-time cooking support, and an emotion engine to maximize the use of ingredients in the home and provide an efficient and enjoyable cooking experience, thereby reducing food waste and promoting healthy eating habits.

[1460] The processing flow will be explained below.

[1461] Image recognition and food ingredient identification

[1462] Step 1:

[1463] The user takes some ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[1464] Step 2:

[1465] The device pre-processes this photo (adjusting resolution, converting format, etc.) and sends it to the server.

[1466] Step 3:

[1467] The server analyzes the received image using an image recognition algorithm (e.g., YOLO, ResNet, etc.) and identifies the ingredients in the photo. Let's say the identified ingredients are tomatoes, chicken, and carrots.

[1468] Step 4:

[1469] The server generates a list of the identified ingredients and returns it to the terminal.

[1470] Process for generating customized recipes

[1471] Step 5:

[1472] The user checks the list of ingredients displayed on the device and enters their preferences and allergy information (e.g., I don't like spicy food).

[1473] Step 6:

[1474] The terminal sends a request to the server along with a list of ingredients based on the entered preferences and allergy information.

[1475] Step 7:

[1476] The server uses a generative AI model (e.g., GPT-4 or Fine-Tuned model) based on the received information to generate a customized recipe, such as a "Tomato and Chicken Stew" recipe using tomatoes and chicken.

[1477] Step 8:

[1478] The server returns the generated customized recipe (ingredients, steps, cooking time, etc.) to the terminal.

[1479] Step 9:

[1480] The device displays the received recipe to the user.

[1481] Real-time cooking support processing

[1482] Step 10:

[1483] If a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and types the question into the device.

[1484] Step 11:

[1485] The device sends the captured photo and the question to the server.

[1486] Step 12:

[1487] The server analyzes the received photos and questions and generates appropriate advice, such as "Please continue baking a little longer."

[1488] Step 13:

[1489] The server transmits the generated advice to the terminal.

[1490] Step 14:

[1491] The device displays the advice to the user.

[1492] Support processing with emotion engine

[1493] Step 15:

[1494] The user inputs emotional information (e.g., feeling stressed or tired) into the device.

[1495] Step 16:

[1496] The terminal transmits the user's emotion information to the server.

[1497] Step 17:

[1498] The server uses an emotion engine to analyze the user's emotional state and provide appropriate support based on that state. For example, a user feeling stressed might be suggested a recipe for a relaxing herbal tea.

[1499] Step 18:

[1500] The server returns the generated cooking instructions and recipes to the terminal.

[1501] Step 19:

[1502] The device displays the received support and recipes to the user.

[1503] In this way, the present invention performs a series of processes to maximize the use of ingredients available to the user and provide an efficient and enjoyable cooking experience. By combining image recognition technology, generative AI models, real-time cooking support, and an emotion engine, it is possible to provide appropriate support and recipe suggestions based on the user's emotional state. This reduces food waste and promotes healthy eating habits.

[1504] Example 2

[1505] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1506] Conventional cooking support systems make it difficult for users to efficiently utilize the individual ingredients they own, and also make it difficult to receive immediate and appropriate support for questions or difficulties that arise during cooking. Furthermore, they are unable to adequately respond to users' emotional state or individual preferences, and are therefore unable to increase satisfaction with cooking. This has led to problems such as wasted ingredients, stress during the cooking process, and a lack of recipes tailored to individual needs.

[1507] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1508] In this invention, the server includes a means for analyzing images using an image recognition algorithm to identify ingredients, a means for generating customized recipes based on the ingredients identified by the generative AI model, and a means for analyzing the user's emotional information using an emotion engine, thereby enabling efficient use of ingredients, real-time cooking support, and recipe provision according to the user's emotional state and preferences.

[1509] "User" refers to an individual who uses the system to receive support in managing ingredients and cooking.

[1510] "Photography device" refers to a device such as a camera or smartphone used to take an image of an ingredient.

[1511] "Image pre-processing" refers to processing such as adjusting the resolution and converting the format of a captured image.

[1512] "Image recognition algorithms" refers to machine learning models and techniques (e.g., YOLO, ResNet) used to identify objects in images.

[1513] "Ingredients" refers to food owned or purchased by the User that is used as an ingredient in cooking.

[1514] "Generative AI Model" refers to an artificial intelligence model (e.g., GPT-4) that generates customized recipes based on identified ingredients.

[1515] "Customized Recipe" refers to a personalized cooking guide generated based on identified ingredients and taking into account the user's preferences and allergy information.

[1516] "Emotion engine" refers to technology that analyzes the user's emotional information and provides appropriate cooking support and recipes based on that state.

[1517] "Real-time cooking support" refers to the function that provides immediate and appropriate advice and solutions to questions or problems that arise while cooking.

[1518] "Preference and Allergy Information" means information related to individual dietary preferences and allergies that a User enters into the System.

[1519] The system of the present invention is designed to maximize the use of ingredients available to users and provide personalized recipes. The system integrates image recognition technology, generative AI models, real-time cooking support functions, and an emotion engine that recognizes user emotions.

[1520] System configuration

[1521] user

[1522] Users take pictures of ingredients they own using a camera such as a smartphone and send them to the device. A smartphone camera is a common camera. If a user has a question or problem while cooking, they can take a picture of the relevant item and enter it as a question into the device. In addition, users can enter information about their preferences, allergies, and emotions into the device.

[1523] Terminal

[1524] The device preprocesses the images taken by the user and sends them to the server. Preprocessing includes adjusting the resolution and converting the format (e.g., from JPEG to PNG). The device displays the received ingredient list and customized recipe, and sends questions and related images from the user while cooking to the server. The device also displays advice and guidance received from the server. Furthermore, the device collects the user's emotional data and sends it to the server.

[1525] server

[1526] The server uses an image recognition algorithm (e.g., YOLO, ResNet) to analyze the image sent by the user and identify the ingredients. Based on the identified ingredient list, the server generates a customized recipe using a generative AI model (e.g., GPT-4). The generated recipe includes ingredients, steps, cooking time, etc. and is sent to the device. The server also analyzes the user's questions and images, generates appropriate advice, and sends it to the device. Furthermore, the server uses an emotion engine to analyze the user's emotional state and provides cooking support and recipes based on the results.

[1527] Specific examples

[1528] Image recognition and food ingredient identification

[1529] 1. The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo with their smartphone camera.

[1530] 2. The device preprocesses the captured photo (e.g., adjusts resolution, converts format) and sends it to the server.

[1531] 3. The server applies an image recognition algorithm to identify the ingredients in the photo. Let's say the ingredients are tomatoes, chicken, and carrots.

[1532] 4. The server generates a list of the identified ingredients and returns it to the terminal.

[1533] Customized recipe generation

[1534] 1. The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[1535] 2. The device sends the ingredient list and the user's preferences and allergy information to the server.

[1536] 3. The server uses the generative AI model based on the received information to generate a customized recipe, for example, a "Chicken and Tomato Stew" recipe.

[1537] 4. The server sends the generated recipe (ingredients, steps, cooking time, etc.) to the terminal.

[1538] 5. The device displays the received recipe to the user.

[1539] Real-time cooking support

[1540] 1. When a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and enters the question into the device.

[1541] 2. The device sends the captured photo and question to the server.

[1542] 3. The server analyzes the received photo and question and generates appropriate advice, such as "Please continue baking a little longer."

[1543] 4. The server sends the generated advice to the terminal.

[1544] 5. The device displays the advice to the user.

[1545] Supported by an emotional engine

[1546] 1. The user inputs emotional information (e.g., feeling stressed or tired) into the device.

[1547] 2. The device sends the user's emotional information to the server.

[1548] 3. The server uses an emotion engine to analyze the user's emotional state and provide appropriate support based on that state. For example, a user feeling stressed might be suggested a recipe for a relaxing herbal tea.

[1549] 4. The server sends the generated cooking instructions and recipes back to the device.

[1550] 5. The device displays the received support and recipes to the user.

[1551] Prompt Sentence Examples

[1552] 1. Image recognition prompt:

[1553] "Please identify the ingredients in this image"

[1554] 2. Recipe generation prompt:

[1555] "Please suggest some non-spicy recipes using these ingredients."

[1556] 3. Cooking support prompt:

[1557] "Please judge whether the chicken in this photo is cooked thoroughly."

[1558] 4. Emotion Engine Support Prompt:

[1559] "Please suggest recipes that will help users relax when they are feeling stressed."

[1560] By using this system, users can use the ingredients they have on hand without waste and enjoy healthy and delicious meals.

[1561] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1562] Step 1:

[1563] The user takes the ingredients they plan to use (e.g., tomatoes, chicken, carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[1564] Input: Images of ingredients in the refrigerator

[1565] Output: Photographed food image (JPEG format)

[1566] Specific actions: Open the camera app on your smartphone, arrange the ingredients, take a photo, and press the shutter button with a "click!"

[1567] Step 2:

[1568] The device pre-processes the captured image, which includes adjusting the resolution and converting the format from JPEG to PNG.

[1569] Input: Photographed food image (JPEG format)

[1570] Output: Preprocessed food image (PNG format)

[1571] Specific operation: Run a program to adjust the resolution and convert the file format to PNG. The resolution adjustment is, for example, to make the width and height 800 pixels each.

[1572] Step 3:

[1573] The terminal sends the preprocessed image to the server and generates a prompt to start the image recognition algorithm.

[1574] Input: Preprocessed food image (PNG format)

[1575] Output: Image data and prompt text are sent to the server

[1576] Specific operation: The preprocessed image is sent to the server via an HTTP request, and a prompt message is sent saying, "Please identify the ingredients from this image."

[1577] Step 4:

[1578] The server applies an image recognition algorithm (e.g., YOLO, ResNet) to analyze the received image.

[1579] Input: Preprocessed food image, prompt

[1580] Output: List of recognized ingredients (e.g., tomato, chicken, carrot)

[1581] What it does: Runs the YOLO algorithm to detect objects in an image and generate an ingredients list. YOLO identifies tomatoes, chicken, and carrots in the image.

[1582] Step 5:

[1583] The server generates a customized recipe using a generative AI model (e.g., GPT-4) based on the identified ingredient list.

[1584] Input: List of recognized ingredients (e.g., tomato, chicken, carrot)

[1585] Output: Customized recipe (ingredients, steps, cooking time, etc.)

[1586] Specific behavior: GPT-4 is fed a list of ingredients and generates a recipe using the prompt "Please suggest a non-spicy recipe using these ingredients." The recipe "Tomato and Chicken Stew" is generated.

[1587] Step 6:

[1588] The server transmits the generated recipe to the terminal.

[1589] Input: Generated custom recipe

[1590] Output: Customized recipe data is sent to the device

[1591] What it does: It uses HTTP push notifications to send the generated recipe information to the device, including the ingredients list, instructions, cooking time, etc.

[1592] Step 7:

[1593] The device displays the received recipe to the user.

[1594] Input: Customized recipe data

[1595] Output: The recipe is displayed on the smartphone screen.

[1596] Specific Behavior: Runs a screen display program to format and display the received recipe information to the user. The UI includes the ingredient list, instructions, and cooking time.

[1597] Step 8:

[1598] If a user has a question while cooking, they can enter it, take a picture of the relevant information, and send it to the device.

[1599] Input: Cooking question or related image

[1600] Output: Questions and image data are sent from the device to the server.

[1601] How it works: You enter your question into your smartphone and take a photo of the related cooking state. The device then sends these to the server.

[1602] Step 9:

[1603] The server analyzes the received question and image and generates appropriate advice.

[1604] Input: Question and associated image data

[1605] Output: Advice (e.g. "Please continue to cook a little longer")

[1606] Specific operation: Analyzes the image using an image analysis algorithm and generates advice based on the question. Appropriate cooking advice is generated in text format.

[1607] Step 10:

[1608] The server transmits the generated advice to the terminal.

[1609] Input: Advice content

[1610] Output: Advice data is sent to the terminal

[1611] Specific operation: Advice information is sent to the device using HTTP push notifications.

[1612] Step 11:

[1613] The device displays the received advice to the user.

[1614] Input: Advice data

[1615] Output: Advice content is displayed on the smartphone screen

[1616] Specific behavior: Run the screen display program, format the received advice information, and display it to the user. The UI includes the advice text.

[1617] Step 12:

[1618] The user inputs their emotional state (e.g., feeling stressed) into the terminal.

[1619] Input: Emotion information

[1620] Output: Emotional information data is sent from the device to the server.

[1621] Specific actions: Enter your emotional state using the multiple choice input form and press the submit button.

[1622] Step 13:

[1623] The server analyzes the emotional information and generates cooking support and relaxing recipes according to the state of the user.

[1624] Input: Emotional information data

[1625] Output: Cooking support and relaxing recipes

[1626] Specific behavior: Activates the emotion engine to analyze the user's emotional state. Based on the analysis results, it generates a relaxing recipe (e.g., "How to make herbal tea").

[1627] Step 14:

[1628] The server transmits the generated cooking instructions and recipes to the terminal.

[1629] Input: Cooking support and relaxing recipes

[1630] Output: Support and recipe data sent to device

[1631] Specific operation: Send cooking support information to the device using HTTP push notifications.

[1632] Step 15:

[1633] The device displays the received support and recipes to the user.

[1634] Input: Cooking support and relaxation recipe data

[1635] Output: Support and recipe information displayed on the smartphone screen

[1636] Specific behavior: Executes the screen display program, formats the received support and recipe information, and displays it to the user. The UI includes support text and relaxation recipe text.

[1637] Through these processing steps, users can receive personalized recipes and real-time cooking support while making the most of the ingredients they have on hand. The system provides efficient ingredient utilization and a comfortable cooking experience.

[1638] (Application example 2)

[1639] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1640] Conventional cooking assistance systems are limited to providing recipes based on the ingredients a user has on hand, and are unable to provide real-time support during cooking or cooking assistance that takes into account the user's emotional state. Furthermore, they are unable to generate recipes that fully reflect the user's preferences and allergies. This has led to problems such as increased stress due to the difficulty users experience while cooking and a lack of suitable recipes.

[1641] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to take images of ingredients owned by the user using a camera, means for receiving the taken images and identifying the ingredients using image recognition technology, means for providing the user with a customized recipe generated based on the identified ingredients, means for generating a customized recipe based on the ingredient list and user information using a generative AI model, and means for inputting the user's emotional information and automatically adjusting the cooking support content provided using an emotion analysis engine. This enables more appropriate and personalized recipes and real-time cooking support that reflect the user's individual needs and preferences, as well as cooking support that responds to the user's emotional state.

[1642] A "photography device" is a device used to take pictures of ingredients owned by a user, such as a smartphone or camera.

[1643] "Image recognition technology" refers to algorithms and techniques for identifying and classifying specific objects or ingredients from photographed images, and includes models such as YOLO and ResNet.

[1644] A "generative AI model" is an artificial intelligence model that generates new text or recipes based on user input or identified data, and examples include GPT-4.

[1645] An "emotion analysis engine" is a technology that analyzes the emotional information entered by the user and provides appropriate cooking support and recipe adjustments based on the results.

[1646] A "customized recipe" is a recipe that meets a user's individual needs and is generated based on identified ingredients, the user's preferences, allergy information, and other individual information.

[1647] "Real-time cooking support" is a service that provides instant solutions and advice to users for any questions or difficulties they may have while cooking.

[1648] "User Information" means information about an individual user, such as preferences, allergy information, and emotional state, that is necessary to provide customized recipes and cooking support.

[1649] The "ingredient list" is a list of ingredients owned by the user, identified using image recognition technology.

[1650] The system embodying this invention integrates image recognition technology, generative AI models, real-time cooking support, and a sentiment analysis engine to recognize user emotions, in order to maximize the use of ingredients on hand and provide personalized recipes.

[1651] System Components

[1652] 1. Users

[1653] Use a camera (such as a smartphone camera) to take a picture of the ingredients you own.

[1654] The captured image is sent to the server via the terminal.

[1655] Take a photo of any questions or problems that arise while cooking and enter them into the device as a question.

[1656] Emotional information, such as stress level or relaxation level, is also entered into the device.

[1657] 2. Terminal

[1658] Images taken by the user are preprocessed and sent to the server.

[1659] Display received ingredient lists and customized recipes to users.

[1660] Collects user emotion data and sends it to the server.

[1661] 3. Server

[1662] Using an image recognition algorithm (e.g., YOLO, ResNet), ingredients are identified from the received image.

[1663] Generate a customized recipe using a generative AI model (e.g., GPT-4) based on the identified list of ingredients.

[1664] An emotion analysis engine that analyzes the user's emotional state is used to adjust the generated recipes and cooking support content.

[1665] Hardware and software used

[1666] Camera: Smartphone camera

[1667] Device: Smartphone or tablet

[1668] Server-side software:

[1669] Image recognition algorithms (e.g., YOLO, ResNet)

[1670] Generative AI models (e.g., GPT-4)

[1671] Sentiment Analysis Engine

[1672] Specific examples

[1673] 1. Image Recognition and Food Identification

[1674] The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[1675] The device preprocesses the photo (adjusting resolution, converting format, etc.) and sends it to the server.

[1676] The server applies an image recognition algorithm to identify the ingredients in the photo, which may be tomatoes, chicken, and carrots.

[1677] 2. Customized recipe generation

[1678] The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[1679] The device sends the ingredient list and the user's preferences and allergy information to the server.

[1680] The server uses the generative AI model to generate customized recipes, such as a "Tomato and Chicken Stew" recipe using tomatoes and chicken.

[1681] An example of a prompt sentence that will generate a recipe is:

[1682] Ingredients: Tomato, Chicken, Carrot

[1683] User Info: {"preferences": ["I hate spicy food"], "allergies": ["nuts"], "emotion_text": "I'm stressed"}

[1684] Recipe:

[1685] 3. Real-time cooking support

[1686] If a user has a question while cooking (e.g., "Is this chicken cooked enough?"), they take a photo of the cooked chicken and type the question into the device.

[1687] The device sends the captured photo and question to the server.

[1688] The server analyzes the received photo and question and generates appropriate advice, such as "Please continue baking a little longer."

[1689] 4. Support with sentiment analysis

[1690] The user's emotional information (e.g., feeling stressed) is input into the device.

[1691] The device transmits the user's emotional information to the server.

[1692] The server uses an emotion analysis engine to analyze the user's emotional state and provides cooking support and recipes based on the results.

[1693] The system allows users to make the most of the ingredients they have and easily obtain personalized recipes. It also responds to questions while cooking in real time and provides cooking support that adapts to their emotional state, making the cooking experience more efficient and enjoyable while reducing food waste.

[1694] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1695] Step 1:

[1696] Users take ingredients from the refrigerator and take a picture of them with their smartphone camera, which then stores the image on the device.

[1697] Step 2:

[1698] The device pre-processes the captured image, adjusts the resolution, and converts it to the appropriate format, then sends the pre-processed image to the server.

[1699] Step 3:

[1700] The server analyzes the received image using an image recognition algorithm (e.g., YOLO, ResNet), and generates a list of ingredients identified from the image and sends it back to the device.

[1701] Step 4:

[1702] Users can review the list of identified ingredients and enter their preferences and allergies into the device, including details such as an aversion to spicy food.

[1703] Step 5:

[1704] The device sends the input user information (ingredients list, preferences, allergy information) to the server, which uses a generative AI model (e.g., GPT-4) to generate a customized recipe based on the input information.

[1705] Step 6:

[1706] The server sends the created customized recipe to the terminal, which displays it to the user and prompts for confirmation.

[1707] Step 7:

[1708] When a user has a question while cooking, they can take a picture of the relevant food and enter it into the device along with their question.

[1709] Step 8:

[1710] The device sends the captured image and question to the server, which analyzes the received data, generates appropriate solutions or advice, and sends it back to the device.

[1711] Step 9:

[1712] The user inputs emotional information (e.g., feeling stressed) into the device, which then transmits this information to the server.

[1713] Step 10:

[1714] The server uses an emotion analysis engine to analyze the user's emotional state, and based on the results, adjusts cooking support and recipe content and sends it back to the device.

[1715] Step 11:

[1716] The device displays the adjusted recipe and cooking support information to the user, who then proceeds with cooking based on this information.

[1717] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1718] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1719] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1720] [Fourth embodiment]

[1721] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1722] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1723] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1724] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1725] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1726] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1727] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1728] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1729] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1730] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1731] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1732] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1733] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1734] System Overview

[1735] The "SmartChef" system of the present invention is a system that maximizes the use of ingredients on hand and provides personalized recipes. The system integrates image recognition technology, generative AI models, and real-time cooking support functions.

[1736] System Components

[1737] 1. Users

[1738] Use a camera (such as a smartphone camera) to take a picture of the ingredients you own.

[1739] The captured image is sent to the server via the terminal.

[1740] Take a photo of any questions or problems that arise while cooking and enter them into the device as a question.

[1741] Enter any necessary preferences or allergy information into the device.

[1742] 2. Terminal

[1743] Images taken by the user are preprocessed and sent to the server.

[1744] Display received ingredient lists and customized recipes to users.

[1745] It receives questions and related images from users while they are cooking and sends them to the server.

[1746] Display advice and guidance received from the server.

[1747] 3. Server

[1748] An image recognition algorithm is used to identify ingredients from the received image.

[1749] A customized recipe is generated based on the identified ingredient list.

[1750] Analyzes user questions and images to generate appropriate advice and solutions.

[1751] The generated ingredient list, customized recipe, and advice are sent back to the device.

[1752] The contents of the customized recipe are adjusted taking into account the user's preferences and allergy information.

[1753] Specific processing flow

[1754] Image recognition and food ingredient identification

[1755] 1. The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo with their smartphone camera.

[1756] 2. The device preprocesses the photo (adjusts resolution, converts format, etc.) and sends it to the server.

[1757] 3. The server receives the submitted image and applies image recognition algorithms to identify the ingredients in the photo, such as tomatoes, chicken, and carrots.

[1758] 4. The server generates a list of the identified ingredients and returns it to the terminal.

[1759] Customized recipe generation

[1760] 1. The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[1761] 2. The terminal sends the ingredient list and the user's preferences and allergy information to the server.

[1762] 3. The server uses the generative AI model based on the received information to generate a customized recipe, such as a "Tomato and Chicken Stew" recipe.

[1763] 4. The server returns the generated recipe (ingredients, steps, cooking time, etc.) to the terminal.

[1764] 5. The device displays the received recipe to the user.

[1765] Real-time cooking support

[1766] 1. When a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and enters the question into the device.

[1767] 2. The device sends the captured photo and question to the server.

[1768] 3. The server analyzes the received photo and question and generates appropriate advice. For example, an AI algorithm determines the doneness of the chicken and generates advice such as "Continue cooking a little longer."

[1769] 4. The server sends the generated advice to the terminal.

[1770] 5. The device displays the advice to the user.

[1771] Educational support and cooking skill development

[1772] 1. The server analyzes data about the user's past cooking history and current skill level.

[1773] 2. The server generates cooking guidance based on the user's current skill level (e.g., how to make a sauce or adjust the heat).

[1774] 3. The server sends the generated guidance to the terminal.

[1775] 4. The device displays the received guidance to the user and provides assistance in improving cooking skills step by step.

[1776] As described above, the "SmartChef" system of the present invention combines image recognition technology, generative AI models, and real-time cooking support to maximize the use of ingredients users have at home and provide an efficient and enjoyable cooking experience, thereby reducing food waste and promoting healthy eating habits.

[1777] The processing flow will be explained below.

[1778] Step 1:

[1779] The user takes some ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[1780] Step 2:

[1781] The device preprocesses the captured photos (adjusting the resolution, converting the format, etc.) and sends them to the server.

[1782] Step 3:

[1783] The server applies an image recognition algorithm (e.g., YOLO, ResNet, etc.) to analyze the received image and identify the ingredients in the photo. Let's say the identified ingredients are tomatoes, chicken, and carrots.

[1784] Step 4:

[1785] The server generates a list of the identified ingredients and returns it to the terminal.

[1786] Step 5:

[1787] The user checks the list of ingredients displayed on the device and enters their preferences and allergy information (e.g., I don't like spicy food).

[1788] Step 6:

[1789] The terminal transmits the ingredient list and the user's preferences and allergy information to the server.

[1790] Step 7:

[1791] The server uses a generative AI model (e.g., GPT-4 or Fine-Tuned model) based on the received information to generate a customized recipe, such as a "Tomato and Chicken Stew" recipe.

[1792] Step 8:

[1793] The server sends the generated recipe (ingredients, steps, cooking time, etc.) back to the terminal.

[1794] Step 9:

[1795] The device displays the received recipe to the user.

[1796] Step 10:

[1797] If a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and types the question into the device.

[1798] Step 11:

[1799] The device sends the captured photo and the question to the server.

[1800] Step 12:

[1801] The server analyzes the received photos and questions and generates appropriate advice, such as "Please continue baking a little longer."

[1802] Step 13:

[1803] The server transmits the generated advice to the terminal.

[1804] Step 14:

[1805] The device displays the advice to the user.

[1806] Step 15:

[1807] The server analyzes data about the user's past cooking history and current skill level.

[1808] Step 16:

[1809] The server generates cooking guidance based on the user's current skill level (e.g., how to make a sauce or adjust the heat).

[1810] Step 17:

[1811] The server transmits the generated guidance to the terminal.

[1812] Step 18:

[1813] The device displays the received guidance to the user, providing assistance in improving cooking skills step by step.

[1814] In this way, the invention combines image recognition technology, generative AI models, and real-time cooking support to maximize the use of ingredients a user has on hand and provide an efficient and enjoyable cooking experience, reducing food waste and promoting healthy eating habits.

[1815] Example 1

[1816] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1817] In modern households, it is often difficult to effectively utilize ingredients available in the refrigerator to prepare delicious and healthy meals. In particular, there is a need for systems that can suggest recipes based on ingredients, resolve cooking questions, and accommodate individual user preferences and allergies. However, conventional systems lack systems that meet these requirements, resulting in increased food waste and cooking stress for users. The present invention aims to solve the above problems and provide users with a convenient and effective cooking assistance system.

[1818] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1819] In this invention, the server includes a means for a user to take images of ingredients owned by the user using a camera, a means for receiving the taken images and identifying the ingredients using image recognition technology, and a means for generating customized recipes using a generative AI model based on the identified ingredients and the user's preferences and allergy information, and providing the recipes to the user, thereby enabling the user to optimally utilize ingredients on hand and receive personalized recipes and cooking support.

[1820] "User" refers to the end user who uses the system to manage ingredients and cook.

[1821] A "photography device" is a device owned by a user for taking pictures of ingredients, and includes, for example, a smartphone camera.

[1822] "Image recognition technology" refers to algorithms and methods for automatically analyzing and identifying ingredients contained in a photographed image.

[1823] "Generative AI Model" refers to an artificial intelligence model used to generate customized recipes based on identified ingredients and user information.

[1824] A "customized recipe" is a recipe that includes individual cooking instructions and an ingredient list that are generated based on the ingredients a user has on hand and that takes into account the user's preferences and allergy information.

[1825] "Cooking Support" is a function that provides appropriate advice and solutions in real time to users when they have questions or difficulties while cooking.

[1826] "Guidance" refers to specific instruction and advice provided to improve a user's cooking skills.

[1827] "Ingredients" are food ingredients that are photographed by the user using a camera and recognized by the system.

[1828] "Preferences and allergy information" refers to information about a user's individual dietary preferences and allergies that the user enters in advance.

[1829] The system of the present invention is designed to maximize the use of ingredients on hand by the user and provide personalized recipes. This system consists of three main components: the user, the terminal, and the server, and their cooperation provides advanced cooking support.

[1830] Overview of the hardware and software used

[1831] 1. User:

[1832] Photography device: Use a device such as a smartphone camera to take pictures of ingredients you own.

[1833] Input device: Use a smartphone or tablet touch screen, keyboard, etc. to input preferences, allergy information, and questions during cooking.

[1834] 2. Device (smartphone, tablet, PC, etc.):

[1835] Image pre-processing software: Adjusts the resolution of received images, converts formats, and removes noise.

[1836] User interface: Displays ingredient lists, customizable recipes, and real-time advice.

[1837] 3. Server:

[1838] Image recognition algorithm: Using libraries such as TensorFlow, ingredients are identified from the received images.

[1839] Generative AI model: Utilizing GPT-3 and other models, it generates customized recipes based on identified ingredients, user preferences, and allergy information.

[1840] Data analysis unit: Analyzes the user's past cooking history and skill level and provides educational guidance.

[1841] Specific processing flow

[1842] Image recognition and food ingredient identification

[1843] 1. User:

[1844] Take some ingredients (e.g. tomatoes, chicken, carrots) out of the refrigerator and take a photo with your smartphone camera.

[1845] 2. Terminal:

[1846] Preprocess the photos (adjust resolution, convert format, etc.) and send them to the server.

[1847] 3. Server:

[1848] The received image is then processed using image recognition algorithms such as TensorFlow to identify ingredients, such as tomatoes, chicken, and carrots.

[1849] The identified ingredient list is returned to the terminal.

[1850] Customized recipe generation

[1851] 1. User:

[1852] Check the list of identified ingredients and enter your preferences and allergy information into the device (e.g., I don't like spicy food).

[1853] 2. Terminal:

[1854] A list of ingredients and information on preferences and allergies is sent to the server.

[1855] 3. Server:

[1856] A generative AI model (e.g., GPT-3) is used to generate a customized recipe, for example, "Chicken and Tomato Stew."

[1857] The generated recipe (ingredients, steps, cooking time, etc.) is sent back to the device.

[1858] 4. Terminal:

[1859] Display the received recipe to the user.

[1860] Real-time cooking support

[1861] 1. User:

[1862] If you have any questions while cooking (e.g., "Is this chicken cooked enough?"), take a photo of the cooking process, enter your question, and send it to the device.

[1863] 2. Terminal:

[1864] Send the question and image to the server.

[1865] 3. Server:

[1866] It uses AI algorithms to analyze the question and generate appropriate advice (e.g., "Please continue cooking a little longer").

[1867] The generated advice is sent to the terminal.

[1868] 4. Terminal:

[1869] Display advice to the user.

[1870] Educational support and cooking skill development

[1871] 1. Server:

[1872] Analyze data about your cooking history and current skill level.

[1873] Generate cooking guidance based on skill level (e.g., how to make a sauce or adjust the heat).

[1874] Send guidance to the device.

[1875] 2. Terminal:

[1876] The received guidance is displayed to the user, providing assistance in improving cooking skills step by step.

[1877] Specific examples

[1878] For example, if a user takes an image of a tomato, chicken, and carrot, the following prompt sentences are input into the generative AI model:

[1879] Example prompt sentence:

[1880] "Generate a non-spicy recipe using tomatoes, chicken, and carrots."

[1881] This prompt will cause the generative AI model to generate a recipe for "Tomato and Chicken Stew" and send it to the device.

[1882] As described above, by providing advanced cooking support through collaboration between the user, device, and server, users can make optimal use of the ingredients they have on hand and receive personalized recipes and cooking support.

[1883] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1884] Step 1:

[1885] The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a picture of the ingredients with the smartphone camera. The user then sends the image to the device via the "SmartChef" app. The input data is the image of the ingredients, and the output data is the image file sent to the device.

[1886] Step 2:

[1887] The terminal preprocesses the received image. Preprocessing includes adjusting the image resolution, format conversion, and noise removal, making the image suitable for analysis. The input data is the image file sent by the user, and the output data is the preprocessed image file.

[1888] Step 3:

[1889] The device sends the preprocessed image file to the server, which uses an image recognition algorithm such as TensorFlow to identify ingredients from the received image. The input data is the preprocessed image file, and the output data is a list of identified ingredients.

[1890] Step 4:

[1891] The server sends the identified ingredient list to the terminal. The terminal displays the ingredient list to the user. The user checks the displayed ingredient list, checks for missing information or errors, and corrects it as necessary. The input data is the ingredient list sent from the server, and the output data is the corrected ingredient list.

[1892] Step 5:

[1893] The user inputs their preferences and allergy information into the terminal. For example, they input "I don't like spicy food." The terminal then sends the input preference and allergy information along with the ingredient list to the server. The input data is the user's preference and allergy information and the ingredient list, and the output data is the information sent to the server.

[1894] Step 6:

[1895] The server generates a customized recipe using a generative AI model (e.g., GPT-3) based on the received information. The input data is the user's preferences, allergy information, and a list of ingredients, and the output data is the generated customized recipe. For example, a recipe called "Tomato and Chicken Stew" is generated.

[1896] Step 7:

[1897] The server sends the generated customized recipe to the terminal. The terminal displays the received recipe to the user. The input data is the generated customized recipe, and the output data is the recipe displayed on the terminal.

[1898] Step 8:

[1899] When a question arises during cooking (e.g., "Is this chicken cooked thoroughly?"), the user takes a photo of the chicken being cooked and enters the question into the terminal and sends it. The input data is the image of the chicken being cooked and the question, and the output data is the data sent to the terminal.

[1900] Step 9:

[1901] The device sends the captured photo and question to the server, which then uses an AI algorithm to analyze the image and question and generate appropriate advice. For example, the advice generated might be, "Please continue cooking a little longer." The input data is the image of the chicken and the question, and the output data is the generated advice.

[1902] Step 10:

[1903] The server sends the generated advice to the terminal. The terminal displays the received advice to the user. The input data is the generated advice, and the output data is the advice displayed on the terminal.

[1904] Step 11:

[1905] The server analyzes data related to the user's cooking history and current skill level and generates guidance for skill improvement. For example, guidance on "how to make a sauce and how to adjust the heat" is generated. The input data is the user's cooking history and skill level, and the output data is the generated guidance.

[1906] Step 12:

[1907] The server transmits the generated guidance to the terminal. The terminal displays the received guidance to the user, providing support for step-by-step improvement of cooking skills. The input data is the generated guidance, and the output data is the guidance displayed on the terminal.

[1908] (Application example 1)

[1909] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1910] In today's busy lifestyles, consumers are expected to prepare healthy, balanced meals while using their time efficiently. However, many difficulties and questions often arise during the process from purchasing ingredients to cooking. In particular, users feel anxious about selecting ingredients, and there is a lack of support that can address questions they have while cooking in real time. This can lead to food waste and make it difficult to prepare healthy meals. To solve these problems, the present invention aims to provide a system that supports the entire process from ingredient selection to cooking in a physical store, allowing users to select ingredients with confidence and cook effectively.

[1911] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1912] In this invention, the server includes means for a user to take images of ingredients owned by the user using a camera, means for receiving the taken images and identifying the ingredients using image recognition technology, means for providing the user with a customized recipe generated based on the identified ingredients, means for inputting the user's preferences and allergy information, means for adjusting the generation of the customized recipe taking into account the input preferences and allergy information, means for scanning ingredients in a physical store and providing an ingredient list and recipe on the spot, and means for resolving questions the user has about ingredient selection in real time. This allows the user to efficiently select ingredients in a physical store and receive real-time support for any questions or difficulties they may have about cooking even after purchase.

[1913] A "user" is a consumer who uses the system, inputs information related to ingredient selection and cooking, and receives feedback.

[1914] A "photography device" is a device owned by the user that includes a camera for taking pictures of ingredients and cooking conditions, and includes smartphones and tablets.

[1915] "Image recognition technology" is a technology that uses computer vision and machine learning algorithms to identify specific ingredients from photographed images.

[1916] A "customized recipe" is a personalized recipe created based on the user's ingredients, preferences, and allergy information.

[1917] "Preference and allergy information" refers to information about the user's food preferences and allergies, and is taken into consideration when generating customized recipes.

[1918] A "physical store" is a physical store where users directly purchase ingredients, such as a supermarket or grocery store.

[1919] "Scanning" is the process by which a user takes a picture of an ingredient in a physical store using a camera, allowing the system to identify the ingredient.

[1920] "Real-time support" is a feature that provides immediate advice and solutions to questions or difficulties users may encounter while selecting ingredients or cooking.

[1921] The "SmartShop" system supports the workflow from ingredient selection to cooking in a brick-and-mortar store. The system consists of a user-specific application (hereafter referred to as a terminal) and a server that performs image recognition and recipe generation.

[1922] System configuration

[1923] The components of this system are as follows:

[1924] 1. Terminal

[1925] Camera: Uses a smartphone or tablet camera to capture the ingredients the user selects in the store and the cooking process.

[1926] User Interface: Displays the ingredient list and customized recipe, and provides a means for users to input preferences and allergy information.

[1927] Communication function: Sends captured images and questions to a server and receives responses from the server.

[1928] 2. Server

[1929] Image Recognition: Uses computer vision and machine learning algorithms to analyze user-submitted images and identify ingredients.

[1930] Generative AI model: Generates customized recipes based on identified ingredients and user preferences and allergies.

[1931] Real-time support function: Generates and provides immediate advice and solutions to any questions or difficulties encountered while cooking.

[1932] Program processing explanation

[1933] Hardware and Software

[1934] Smartphone / Tablet (e.g. iPhone, Samsung Galaxy, etc.): Serves as a user interface and capture device.

[1935] Server: Use a cloud-based server such as Amazon Web Services (AWS) or Google Cloud Platform (GCP).

[1936] Python + Flask: Used to build the backend API.

[1937] PIL (Python Imaging Library) and numpy: Used for image and data processing.

[1938] Computer vision technology: OpenCV and TensorFlow are used as algorithms for food ingredient recognition.

[1939] Generative AI models: Use generative AI, such as OpenAI's GPT-3, to generate customized recipes.

[1940] Data processing and calculation

[1941] The device takes pictures of ingredients and cooking conditions using the smartphone camera and sends the image data to the server, where it is pre-processed by adjusting the resolution and converting the format.

[1942] The server analyzes the received image using computer vision technology to identify the ingredients, using neural network models for ingredient identification (e.g., ResNet, YOLO, etc.).

[1943] The generative AI model takes the identified ingredient list and the user's preferences and allergy information as inputs to generate customized recipes, providing recipes tailored to the user's needs.

[1944] The real-time support function analyzes captured images and questions that users may have while cooking, and generates appropriate advice using a generative AI model, which is then sent back to the device.

[1945] Specific examples

[1946] Example 1: Use in a physical store

[1947] A user scans "tomatoes, chicken, and carrots" in a physical store using their smartphone. Based on this information and the user's pre-entered preferences (such as not liking spicy food), the system provides a recipe for "tomato and chicken stew." If the user has questions about how to check the freshness of tomatoes at the time of purchase, the server immediately returns advice.

[1948] Example prompt:

[1949] Ingredients: Tomato, chicken, carrot. Preferences: Don't like spicy food. Allergies: None. Please recommend some recipes using these ingredients.

[1950] This allows users to smoothly select ingredients and cook them, reducing food waste and providing an efficient and enjoyable cooking experience.

[1951] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1952] Step 1:

[1953] The user takes a photo of the ingredients they plan to purchase in a physical store using the smartphone camera. The user then takes a picture of the ingredients and saves the image on the device through the application. The input is the photographed image of the ingredients, and the output is preprocessed image data.

[1954] Step 2:

[1955] The device preprocesses the captured images and sends them to the server. Preprocessing includes image resolution adjustment, format conversion, noise removal, etc. The input is the captured raw image data, and the output is the preprocessed image data.

[1956] Step 3:

[1957] The server analyzes the received image data and identifies the ingredients using image recognition technology. Specifically, it uses computer vision algorithms (e.g., YOLO, ResNet) to identify the type of ingredients. The input is the preprocessed image data, and the output is a list of identified ingredients.

[1958] Step 4:

[1959] The server generates a list of identified ingredients and returns it to the device. The user can then check the list on the device. The input is the ingredient information identified by image recognition, and the output is the ingredient list.

[1960] Step 5:

[1961] The user inputs their preferences and allergy information into the terminal. The input is the user's preferences and allergies, and the output is the updated information.

[1962] Step 6:

[1963] The terminal transmits the ingredient list and the user's preferences and allergy information to the server. The input is the user input information and the specified ingredient list, and the output is the data sent to the server.

[1964] Step 7:

[1965] The server generates a customized recipe using a generative AI model based on the received information. The AI ​​model uses OpenAI's GPT-3 and other models to automatically generate recipes that take into account the identified ingredients and the user's preferences and allergy information. The input is the identified ingredient list and the user's preference data, and the output is a customized recipe.

[1966] Step 8:

[1967] The server returns the generated customized recipe to the terminal, which then displays the received recipe to the user. The input is the generated recipe information, and the output is the recipe displayed on the user interface.

[1968] Step 9:

[1969] If a user has a question or difficulty while cooking, they can take a photo and input the question into the device. The input is the cooking question and related images, and the output is the question data stored on the device.

[1970] Step 10:

[1971] The terminal sends the user's question and related images to the server. The input is the question data from the user and the related images, and the output is the data sent to the server.

[1972] Step 11:

[1973] The server analyzes the received questions and images and generates appropriate advice and solutions. It again uses a generative AI model to derive a specific solution to the question or problem. The input is the user's question data and related images, and the output is the generated advice.

[1974] Step 12:

[1975] The server sends the generated advice back to the terminal, which displays it to the user. The input is the advice data, and the output is the advice displayed in the user interface.

[1976] This process allows users to efficiently select ingredients in-store and receive real-time support for any cooking questions or difficulties after purchase.

[1977] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1978] System Overview

[1979] The system of the present invention, which aims to maximize the use of ingredients on hand and provide personalized recipes, integrates image recognition technology, generative AI models, real-time cooking support, and an emotion engine that recognizes user emotions.

[1980] System Components

[1981] 1. Users

[1982] Use a camera (such as a smartphone camera) to take a picture of the ingredients you own.

[1983] The captured image is sent to the server via the terminal.

[1984] Take a photo of any questions or problems that arise while cooking and enter them into the device as a question.

[1985] Enter any necessary preferences, allergies, and emotional information into the device.

[1986] The system provides cooking guidance based on skill level and relaxing recipes.

[1987] 2. Terminal

[1988] Images taken by the user are preprocessed and sent to the server.

[1989] Display received ingredient lists and customized recipes to users.

[1990] It receives questions and related images from users while they are cooking and sends them to the server.

[1991] Display advice and guidance received from the server.

[1992] Collects user emotion data and sends it to the server.

[1993] Providing cooking support based on the user's emotional state.

[1994] 3. Server

[1995] An image recognition algorithm is used to identify ingredients from the received image.

[1996] A customized recipe is generated based on the identified ingredient list.

[1997] Analyzes user questions and images to generate appropriate advice and solutions.

[1998] The generated ingredient list, customized recipe, and advice are sent back to the device.

[1999] The contents of the customized recipe are adjusted taking into account the user's preferences and allergy information.

[2000] It uses an emotion engine to analyze the user's emotional state and provides cooking support and recipes based on the results.

[2001] Program processing and specific examples

[2002] Image recognition and food ingredient identification

[2003] 1. The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo with their smartphone camera.

[2004] 2. The device preprocesses the photo (adjusts resolution, converts format, etc.) and sends it to the server.

[2005] 3. The server applies an image recognition algorithm (e.g., YOLO, ResNet, etc.) to analyze the submitted image and identify the ingredients in the photo. Let's say the identified ingredients are tomatoes, chicken, and carrots.

[2006] 4. The server generates a list of the identified ingredients and returns it to the terminal.

[2007] Customized recipe generation

[2008] 1. The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[2009] 2. The device sends the ingredient list and the user's preferences and allergy information to the server.

[2010] 3. The server uses a generative AI model (e.g., GPT-4 or Fine-Tuned model) based on the received information to generate a customized recipe. For example, a recipe for "Tomato and Chicken Stew" using tomatoes and chicken is created.

[2011] 4. The server returns the generated recipe (ingredients, steps, cooking time, etc.) to the terminal.

[2012] 5. The device displays the received recipe to the user.

[2013] Real-time cooking support

[2014] 1. When a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and enters the question into the device.

[2015] 2. The device sends the captured photo and question to the server.

[2016] 3. The server analyzes the received photo and question and generates appropriate advice, such as "Please continue baking a little longer."

[2017] 4. The server sends the generated advice to the terminal.

[2018] 5. The device displays the advice to the user.

[2019] Supported by an emotional engine

[2020] 1. The user inputs emotional information (e.g., feeling stressed or tired) into the device.

[2021] 2. The device sends the user's emotional information to the server.

[2022] 3. The server uses an emotion engine to analyze the user's emotional state and provide appropriate support based on that state. For example, a user feeling stressed might be suggested a recipe for a relaxing herbal tea.

[2023] 4. The server sends the generated cooking instructions and recipes back to the device.

[2024] 5. The device displays the received support and recipes to the user.

[2025] As described above, the system of the present invention combines image recognition technology, generative AI models, real-time cooking support, and an emotion engine to maximize the use of ingredients in the home and provide an efficient and enjoyable cooking experience, thereby reducing food waste and promoting healthy eating habits.

[2026] The processing flow will be explained below.

[2027] Image recognition and food ingredient identification

[2028] Step 1:

[2029] The user takes some ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[2030] Step 2:

[2031] The device pre-processes this photo (adjusting resolution, converting format, etc.) and sends it to the server.

[2032] Step 3:

[2033] The server analyzes the received image using an image recognition algorithm (e.g., YOLO, ResNet, etc.) and identifies the ingredients in the photo. Let's say the identified ingredients are tomatoes, chicken, and carrots.

[2034] Step 4:

[2035] The server generates a list of the identified ingredients and returns it to the terminal.

[2036] Process for generating customized recipes

[2037] Step 5:

[2038] The user checks the list of ingredients displayed on the device and enters their preferences and allergy information (e.g., I don't like spicy food).

[2039] Step 6:

[2040] The terminal sends a request to the server along with a list of ingredients based on the entered preferences and allergy information.

[2041] Step 7:

[2042] The server uses a generative AI model (e.g., GPT-4 or Fine-Tuned model) based on the received information to generate a customized recipe, such as a "Tomato and Chicken Stew" recipe using tomatoes and chicken.

[2043] Step 8:

[2044] The server returns the generated customized recipe (ingredients, steps, cooking time, etc.) to the terminal.

[2045] Step 9:

[2046] The device displays the received recipe to the user.

[2047] Real-time cooking support processing

[2048] Step 10:

[2049] If a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and types the question into the device.

[2050] Step 11:

[2051] The device sends the captured photo and the question to the server.

[2052] Step 12:

[2053] The server analyzes the received photos and questions and generates appropriate advice, such as "Please continue baking a little longer."

[2054] Step 13:

[2055] The server transmits the generated advice to the terminal.

[2056] Step 14:

[2057] The device displays the advice to the user.

[2058] Support processing with emotion engine

[2059] Step 15:

[2060] The user inputs emotional information (e.g., feeling stressed or tired) into the device.

[2061] Step 16:

[2062] The terminal transmits the user's emotion information to the server.

[2063] Step 17:

[2064] The server uses an emotion engine to analyze the user's emotional state and provide appropriate support based on that state. For example, a user feeling stressed might be suggested a recipe for a relaxing herbal tea.

[2065] Step 18:

[2066] The server returns the generated cooking instructions and recipes to the terminal.

[2067] Step 19:

[2068] The device displays the received support and recipes to the user.

[2069] In this way, the present invention performs a series of processes to maximize the use of ingredients available to the user and provide an efficient and enjoyable cooking experience. By combining image recognition technology, generative AI models, real-time cooking support, and an emotion engine, it is possible to provide appropriate support and recipe suggestions based on the user's emotional state. This reduces food waste and promotes healthy eating habits.

[2070] Example 2

[2071] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2072] Conventional cooking support systems make it difficult for users to efficiently utilize the individual ingredients they own, and also make it difficult to receive immediate and appropriate support for questions or difficulties that arise during cooking. Furthermore, they are unable to adequately respond to users' emotional state or individual preferences, and are therefore unable to increase satisfaction with cooking. This has led to problems such as wasted ingredients, stress during the cooking process, and a lack of recipes tailored to individual needs.

[2073] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2074] In this invention, the server includes a means for analyzing images using an image recognition algorithm to identify ingredients, a means for generating customized recipes based on the ingredients identified by the generative AI model, and a means for analyzing the user's emotional information using an emotion engine, thereby enabling efficient use of ingredients, real-time cooking support, and recipe provision according to the user's emotional state and preferences.

[2075] "User" refers to an individual who uses the system to receive support in managing ingredients and cooking.

[2076] "Photography device" refers to a device such as a camera or smartphone used to take an image of an ingredient.

[2077] "Image pre-processing" refers to processing such as adjusting the resolution and converting the format of a captured image.

[2078] "Image recognition algorithms" refers to machine learning models and techniques (e.g., YOLO, ResNet) used to identify objects in images.

[2079] "Ingredients" refers to food owned or purchased by the User that is used as an ingredient in cooking.

[2080] "Generative AI Model" refers to an artificial intelligence model (e.g., GPT-4) that generates customized recipes based on identified ingredients.

[2081] "Customized Recipe" refers to a personalized cooking guide generated based on identified ingredients and taking into account the user's preferences and allergy information.

[2082] "Emotion engine" refers to technology that analyzes the user's emotional information and provides appropriate cooking support and recipes based on that state.

[2083] "Real-time cooking support" refers to the function that provides immediate and appropriate advice and solutions to questions or problems that arise while cooking.

[2084] "Preference and Allergy Information" means information related to individual dietary preferences and allergies that a User enters into the System.

[2085] The system of the present invention is designed to maximize the use of ingredients available to users and provide personalized recipes. The system integrates image recognition technology, generative AI models, real-time cooking support functions, and an emotion engine that recognizes user emotions.

[2086] System configuration

[2087] user

[2088] Users take pictures of ingredients they own using a camera such as a smartphone and send them to the device. A smartphone camera is a common camera. If a user has a question or problem while cooking, they can take a picture of the relevant item and enter it as a question into the device. In addition, users can enter information about their preferences, allergies, and emotions into the device.

[2089] Terminal

[2090] The device preprocesses the images taken by the user and sends them to the server. Preprocessing includes adjusting the resolution and converting the format (e.g., from JPEG to PNG). The device displays the received ingredient list and customized recipe, and sends questions and related images from the user while cooking to the server. The device also displays advice and guidance received from the server. Furthermore, the device collects the user's emotional data and sends it to the server.

[2091] server

[2092] The server uses an image recognition algorithm (e.g., YOLO, ResNet) to analyze the image sent by the user and identify the ingredients. Based on the identified ingredient list, the server generates a customized recipe using a generative AI model (e.g., GPT-4). The generated recipe includes ingredients, steps, cooking time, etc. and is sent to the device. The server also analyzes the user's questions and images, generates appropriate advice, and sends it to the device. Furthermore, the server uses an emotion engine to analyze the user's emotional state and provides cooking support and recipes based on the results.

[2093] Specific examples

[2094] Image recognition and food ingredient identification

[2095] 1. The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo with their smartphone camera.

[2096] 2. The device preprocesses the captured photo (e.g., adjusts resolution, converts format) and sends it to the server.

[2097] 3. The server applies an image recognition algorithm to identify the ingredients in the photo. Let's say the ingredients are tomatoes, chicken, and carrots.

[2098] 4. The server generates a list of the identified ingredients and returns it to the terminal.

[2099] Customized recipe generation

[2100] 1. The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[2101] 2. The device sends the ingredient list and the user's preferences and allergy information to the server.

[2102] 3. The server uses the generative AI model based on the received information to generate a customized recipe, for example, a "Chicken and Tomato Stew" recipe.

[2103] 4. The server sends the generated recipe (ingredients, steps, cooking time, etc.) to the terminal.

[2104] 5. The device displays the received recipe to the user.

[2105] Real-time cooking support

[2106] 1. When a question arises during cooking (e.g., "Is this chicken cooked enough?"), the user takes a photo of the cooked chicken and enters the question into the device.

[2107] 2. The device sends the captured photo and question to the server.

[2108] 3. The server analyzes the received photo and question and generates appropriate advice, such as "Please continue baking a little longer."

[2109] 4. The server sends the generated advice to the terminal.

[2110] 5. The device displays the advice to the user.

[2111] Supported by an emotional engine

[2112] 1. The user inputs emotional information (e.g., feeling stressed or tired) into the device.

[2113] 2. The device sends the user's emotional information to the server.

[2114] 3. The server uses an emotion engine to analyze the user's emotional state and provide appropriate support based on that state. For example, a user feeling stressed might be suggested a recipe for a relaxing herbal tea.

[2115] 4. The server sends the generated cooking instructions and recipes back to the device.

[2116] 5. The device displays the received support and recipes to the user.

[2117] Prompt Sentence Examples

[2118] 1. Image recognition prompt:

[2119] "Please identify the ingredients in this image"

[2120] 2. Recipe generation prompt:

[2121] "Please suggest some non-spicy recipes using these ingredients."

[2122] 3. Cooking support prompt:

[2123] "Please judge whether the chicken in this photo is cooked thoroughly."

[2124] 4. Emotion Engine Support Prompt:

[2125] "Please suggest recipes that will help users relax when they are feeling stressed."

[2126] By using this system, users can use the ingredients they have on hand without waste and enjoy healthy and delicious meals.

[2127] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2128] Step 1:

[2129] The user takes the ingredients they plan to use (e.g., tomatoes, chicken, carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[2130] Input: Images of ingredients in the refrigerator

[2131] Output: Photographed food image (JPEG format)

[2132] Specific actions: Open the camera app on your smartphone, arrange the ingredients, take a photo, and press the shutter button with a "click!"

[2133] Step 2:

[2134] The device pre-processes the captured image, which includes adjusting the resolution and converting the format from JPEG to PNG.

[2135] Input: Photographed food image (JPEG format)

[2136] Output: Preprocessed food image (PNG format)

[2137] Specific operation: Run a program to adjust the resolution and convert the file format to PNG. The resolution adjustment is, for example, to make the width and height 800 pixels each.

[2138] Step 3:

[2139] The terminal sends the preprocessed image to the server and generates a prompt to start the image recognition algorithm.

[2140] Input: Preprocessed food image (PNG format)

[2141] Output: Image data and prompt text are sent to the server

[2142] Specific operation: The preprocessed image is sent to the server via an HTTP request, and a prompt message is sent saying, "Please identify the ingredients from this image."

[2143] Step 4:

[2144] The server applies an image recognition algorithm (e.g., YOLO, ResNet) to analyze the received image.

[2145] Input: Preprocessed food image, prompt

[2146] Output: List of recognized ingredients (e.g., tomato, chicken, carrot)

[2147] What it does: Runs the YOLO algorithm to detect objects in an image and generate an ingredients list. YOLO identifies tomatoes, chicken, and carrots in the image.

[2148] Step 5:

[2149] The server generates a customized recipe using a generative AI model (e.g., GPT-4) based on the identified ingredient list.

[2150] Input: List of recognized ingredients (e.g., tomato, chicken, carrot)

[2151] Output: Customized recipe (ingredients, steps, cooking time, etc.)

[2152] Specific behavior: GPT-4 is fed a list of ingredients and generates a recipe using the prompt "Please suggest a non-spicy recipe using these ingredients." The recipe "Tomato and Chicken Stew" is generated.

[2153] Step 6:

[2154] The server transmits the generated recipe to the terminal.

[2155] Input: Generated custom recipe

[2156] Output: Customized recipe data is sent to the device

[2157] What it does: It uses HTTP push notifications to send the generated recipe information to the device, including the ingredients list, instructions, cooking time, etc.

[2158] Step 7:

[2159] The device displays the received recipe to the user.

[2160] Input: Customized recipe data

[2161] Output: The recipe is displayed on the smartphone screen.

[2162] Specific Behavior: Runs a screen display program to format and display the received recipe information to the user. The UI includes the ingredient list, instructions, and cooking time.

[2163] Step 8:

[2164] If a user has a question while cooking, they can enter it, take a picture of the relevant information, and send it to the device.

[2165] Input: Cooking question or related image

[2166] Output: Questions and image data are sent from the device to the server.

[2167] How it works: You enter your question into your smartphone and take a photo of the related cooking state. The device then sends these to the server.

[2168] Step 9:

[2169] The server analyzes the received question and image and generates appropriate advice.

[2170] Input: Question and associated image data

[2171] Output: Advice (e.g. "Please continue to cook a little longer")

[2172] Specific operation: Analyzes the image using an image analysis algorithm and generates advice based on the question. Appropriate cooking advice is generated in text format.

[2173] Step 10:

[2174] The server transmits the generated advice to the terminal.

[2175] Input: Advice content

[2176] Output: Advice data is sent to the terminal

[2177] Specific operation: Advice information is sent to the device using HTTP push notifications.

[2178] Step 11:

[2179] The device displays the received advice to the user.

[2180] Input: Advice data

[2181] Output: Advice content is displayed on the smartphone screen

[2182] Specific behavior: Run the screen display program, format the received advice information, and display it to the user. The UI includes the advice text.

[2183] Step 12:

[2184] The user inputs their emotional state (e.g., feeling stressed) into the terminal.

[2185] Input: Emotion information

[2186] Output: Emotional information data is sent from the device to the server.

[2187] Specific actions: Enter your emotional state using the multiple choice input form and press the submit button.

[2188] Step 13:

[2189] The server analyzes the emotional information and generates cooking support and relaxing recipes according to the state of the user.

[2190] Input: Emotional information data

[2191] Output: Cooking support and relaxing recipes

[2192] Specific behavior: Activates the emotion engine to analyze the user's emotional state. Based on the analysis results, it generates a relaxing recipe (e.g., "How to make herbal tea").

[2193] Step 14:

[2194] The server transmits the generated cooking instructions and recipes to the terminal.

[2195] Input: Cooking support and relaxing recipes

[2196] Output: Support and recipe data sent to device

[2197] Specific operation: Send cooking support information to the device using HTTP push notifications.

[2198] Step 15:

[2199] The device displays the received support and recipes to the user.

[2200] Input: Cooking support and relaxation recipe data

[2201] Output: Support and recipe information displayed on the smartphone screen

[2202] Specific behavior: Executes the screen display program, formats the received support and recipe information, and displays it to the user. The UI includes support text and relaxation recipe text.

[2203] Through these processing steps, users can receive personalized recipes and real-time cooking support while making the most of the ingredients they have on hand. The system provides efficient ingredient utilization and a comfortable cooking experience.

[2204] (Application example 2)

[2205] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2206] Conventional cooking assistance systems are limited to providing recipes based on the ingredients a user has on hand, and are unable to provide real-time support during cooking or cooking assistance that takes into account the user's emotional state. Furthermore, they are unable to generate recipes that fully reflect the user's preferences and allergies. This has led to problems such as increased stress due to the difficulty users experience while cooking and a lack of suitable recipes.

[2207] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to take images of ingredients owned by the user using a camera, means for receiving the taken images and identifying the ingredients using image recognition technology, means for providing the user with a customized recipe generated based on the identified ingredients, means for generating a customized recipe based on the ingredient list and user information using a generative AI model, and means for inputting the user's emotional information and automatically adjusting the cooking support content provided using an emotion analysis engine. This enables more appropriate and personalized recipes and real-time cooking support that reflect the user's individual needs and preferences, as well as cooking support that responds to the user's emotional state.

[2208] A "photography device" is a device used to take pictures of ingredients owned by a user, such as a smartphone or camera.

[2209] "Image recognition technology" refers to algorithms and techniques for identifying and classifying specific objects or ingredients from photographed images, and includes models such as YOLO and ResNet.

[2210] A "generative AI model" is an artificial intelligence model that generates new text or recipes based on user input or identified data, and examples include GPT-4.

[2211] An "emotion analysis engine" is a technology that analyzes the emotional information entered by the user and provides appropriate cooking support and recipe adjustments based on the results.

[2212] A "customized recipe" is a recipe that meets a user's individual needs and is generated based on identified ingredients, the user's preferences, allergy information, and other individual information.

[2213] "Real-time cooking support" is a service that provides instant solutions and advice to users for any questions or difficulties they may have while cooking.

[2214] "User Information" means information about an individual user, such as preferences, allergy information, and emotional state, that is necessary to provide customized recipes and cooking support.

[2215] The "ingredient list" is a list of ingredients owned by the user, identified using image recognition technology.

[2216] The system embodying this invention integrates image recognition technology, generative AI models, real-time cooking support, and a sentiment analysis engine to recognize user emotions, in order to maximize the use of ingredients on hand and provide personalized recipes.

[2217] System Components

[2218] 1. Users

[2219] Use a camera (such as a smartphone camera) to take a picture of the ingredients you own.

[2220] The captured image is sent to the server via the terminal.

[2221] Take a photo of any questions or problems that arise while cooking and enter them into the device as a question.

[2222] Emotional information, such as stress level or relaxation level, is also entered into the device.

[2223] 2. Terminal

[2224] Images taken by the user are preprocessed and sent to the server.

[2225] Display received ingredient lists and customized recipes to users.

[2226] Collects user emotion data and sends it to the server.

[2227] 3. Server

[2228] Using an image recognition algorithm (e.g., YOLO, ResNet), ingredients are identified from the received image.

[2229] Generate a customized recipe using a generative AI model (e.g., GPT-4) based on the identified list of ingredients.

[2230] An emotion analysis engine that analyzes the user's emotional state is used to adjust the generated recipes and cooking support content.

[2231] Hardware and software used

[2232] Camera: Smartphone camera

[2233] Device: Smartphone or tablet

[2234] Server-side software:

[2235] Image recognition algorithms (e.g., YOLO, ResNet)

[2236] Generative AI models (e.g., GPT-4)

[2237] Sentiment Analysis Engine

[2238] Specific examples

[2239] 1. Image Recognition and Food Identification

[2240] The user takes ingredients (e.g., tomatoes, chicken, and carrots) from the refrigerator and takes a photo of them with their smartphone camera.

[2241] The device preprocesses the photo (adjusting resolution, converting format, etc.) and sends it to the server.

[2242] The server applies an image recognition algorithm to identify the ingredients in the photo, which may be tomatoes, chicken, and carrots.

[2243] 2. Customized recipe generation

[2244] The user checks the list of identified ingredients and enters their preferences and allergy information into the device (e.g., I don't like spicy food).

[2245] The device sends the ingredient list and the user's preferences and allergy information to the server.

[2246] The server uses the generative AI model to generate customized recipes, such as a "Tomato and Chicken Stew" recipe using tomatoes and chicken.

[2247] An example of a prompt sentence that will generate a recipe is:

[2248] Ingredients: Tomato, Chicken, Carrot

[2249] User Info: {"preferences": ["I hate spicy food"], "allergies": ["nuts"], "emotion_text": "I'm stressed"}

[2250] Recipe:

[2251] 3. Real-time cooking support

[2252] If a user has a question while cooking (e.g., "Is this chicken cooked enough?"), they take a photo of the cooked chicken and type the question into the device.

[2253] The device sends the captured photo and question to the server.

[2254] The server analyzes the received photo and question and generates appropriate advice, such as "Please continue baking a little longer."

[2255] 4. Support with sentiment analysis

[2256] The user's emotional information (e.g., feeling stressed) is input into the device.

[2257] The device transmits the user's emotional information to the server.

[2258] The server uses an emotion analysis engine to analyze the user's emotional state and provides cooking support and recipes based on the results.

[2259] The system allows users to make the most of the ingredients they have and easily obtain personalized recipes. It also responds to questions while cooking in real time and provides cooking support that adapts to their emotional state, making the cooking experience more efficient and enjoyable while reducing food waste.

[2260] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2261] Step 1:

[2262] Users take ingredients from the refrigerator and take a picture of them with their smartphone camera, which then stores the image on the device.

[2263] Step 2:

[2264] The device pre-processes the captured image, adjusts the resolution, and converts it to the appropriate format, then sends the pre-processed image to the server.

[2265] Step 3:

[2266] The server analyzes the received image using an image recognition algorithm (e.g., YOLO, ResNet), and generates a list of ingredients identified from the image and sends it back to the device.

[2267] Step 4:

[2268] Users can review the list of identified ingredients and enter their preferences and allergies into the device, including details such as an aversion to spicy food.

[2269] Step 5:

[2270] The device sends the input user information (ingredients list, preferences, allergy information) to the server, which uses a generative AI model (e.g., GPT-4) to generate a customized recipe based on the input information.

[2271] Step 6:

[2272] The server sends the created customized recipe to the terminal, which displays it to the user and prompts for confirmation.

[2273] Step 7:

[2274] When a user has a question while cooking, they can take a picture of the relevant food and enter it into the device along with their question.

[2275] Step 8:

[2276] The device sends the captured image and question to the server, which analyzes the received data, generates appropriate solutions or advice, and sends it back to the device.

[2277] Step 9:

[2278] The user inputs emotional information (e.g., feeling stressed) into the device, which then transmits this information to the server.

[2279] Step 10:

[2280] The server uses an emotion analysis engine to analyze the user's emotional state, and based on the results, adjusts cooking support and recipe content and sends it back to the device.

[2281] Step 11:

[2282] The device displays the adjusted recipe and cooking support information to the user, who then proceeds with cooking based on this information.

[2283] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2284] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2285] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2286] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2287] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2288] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2289] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2290] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2291] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2292] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2293] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2294] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2295] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2296] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2297] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2298] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2299] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2300] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2301] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2302] The above-described description and illustrations are a detailed explanation of the par...

Claims

1. A means for a user to take an image of an ingredient they own using a photographing device; A means for receiving the captured image and identifying the food ingredient using image recognition technology; a means for providing the user with a customized recipe generated based on the identified ingredients; A system including:

2. A means for a user to send questions and related images in response to questions or difficulties that arise during cooking; A means for analyzing the submitted questions and images, generating appropriate solutions and advice, and providing them to the user; The system of claim 1 further comprising:

3. a means for inputting user preferences and allergy information; a means for adjusting the generation of customized recipes taking into account input preference and allergy information; The system of claim 1 further comprising:

Images