System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting food ingredient information; artificial intelligence means for analyzing the input food ingredient information and generating a candidate cooking recipe; means for generating an animation of a cooking procedure based on the generated cooking recipe; user interface means for displaying the generated cooking recipe and the cooking procedure animation; and electronic commerce means for purchasing necessary additional food ingredients.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In everyday cooking, many families and people living alone struggle with making the same dishes over and over again because they don't know how to make effective use of leftover ingredients in the refrigerator, leading to problems of wasting food and becoming bored with their cooking. Furthermore, when trying new recipes, they face the challenge of finding the necessary additional ingredients and finding the specific cooking steps for each recipe. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes a means for inputting ingredient information, an artificial intelligence means for analyzing the input ingredient information and generating candidate recipes, a means for generating videos of cooking steps based on the generated recipes, a user interface means for displaying the generated recipes and videos of cooking steps, and an e-commerce means for purchasing additional ingredients. This system not only allows users to effectively use ingredients in their refrigerators, but also makes it easy to procure additional ingredients and increase the variety of dishes they can make. Furthermore, because specific cooking steps can be viewed in videos, even beginners can confidently try new recipes.

[0006] The "means for inputting ingredient information" refers to a device or program that provides an interface for the user to input information about ingredients stored in the refrigerator.

[0007] The "artificial intelligence means" is a system that has the function of analyzing input ingredient information and generating appropriate cooking recipes based on that information.

[0008] The "means for generating video" refers to a device or program that generates a video that visually displays cooking steps based on the generated cooking recipe.

[0009] The "user interface means" is a device or program that provides an interface for displaying the generated cooking recipe and cooking procedure video to the user.

[0010] An "electronic commerce instrument" is a device or program that provides a platform for a user to purchase additional ingredients online. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0012] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0013] First, the terms used in the following description will be explained.

[0014] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0015] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0016] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0017] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0019] [First embodiment]

[0020] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0021] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0024] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0027] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0031] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0032] The present invention is a system that proposes cooking recipes using ingredients in a user's refrigerator and generates cooking videos for those recipes. This system allows users to efficiently create new cooking menus. It also allows users to easily purchase additional ingredients as needed to increase the variety of dishes. The specific program processing of this system is described below.

[0033] System Overview

[0034] 1. Ingredient input method: The user launches the application on their smartphone or PC and inputs the ingredients they have in their refrigerator. This ingredient information is converted to JSON format by the device and sent to the server.

[0035] 2. Recipe generation: The server analyzes the received ingredient information and passes the data to the generation AI to generate a cooking recipe. For example, based on ingredient information such as "chicken, cabbage, and carrots," the generation AI generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[0036] 3. Video generation means: The server provides the generated recipe information to the video generation AI, which generates a video showing the cooking steps in a visually easy-to-understand format. The generated video file is returned to the server.

[0037] 4. User interface: The server sends the acquired recipe information and the generated video to the user's device and displays them through the user interface. The user can view the suggested recipes and videos and select their favorite dishes.

[0038] 5. E-commerce method: If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The user is notified of the delivery status in real time.

[0039] Specific examples

[0040] As a specific example, consider the case where a user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator.

[0041] 1. User: Opens the app on his smartphone and enters the ingredients in his refrigerator: "chicken," "cabbage," and "carrots."

[0042] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[0043] 3. Server: Analyzes the received ingredient list, inputs "chicken, cabbage, carrots" into the generation AI, and generates the following recipes ("stir-fried chicken" and "cabbage and carrot salad").

[0044] 4. Video generation method: Request the video generation AI to generate a cooking video for the above recipe. The generated video file is returned to the server.

[0045] 5. Server: Sends the generated recipes and videos to the user's device.

[0046] 6. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[0047] 7. User: Review suggested recipes and videos and select "Chicken Stir-fry."

[0048] 8. E-commerce method: If the user does not have the necessary additional ingredients (e.g., soy sauce), the user requests to purchase the additional ingredients through the app. The device sends the list of ingredients to the e-commerce site.

[0049] 9. Server: Retrieves soy sauce inventory and price information from the e-commerce site's API and presents it to the user.

[0050] 10. User: Confirms purchase.

[0051] 11. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[0052] 12. E-commerce: Process orders and initiate logistics processes in a short time.

[0053] 13. Server: Notifies the user of delivery status in real time.

[0054] 14. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[0055] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, reduces food waste, and improves the user experience.

[0056] The processing flow will be explained below.

[0057] Step 1:

[0058] User: Launches the application on their smartphone or PC and accesses the screen where they can input the ingredients in their refrigerator.

[0059] Step 2:

[0060] User: Enter the ingredients they currently have on hand, such as "chicken," "cabbage," and "carrots," into the text field.

[0061] Step 3:

[0062] Terminal: Convert the input ingredient list into JSON format and send it to the server.

[0063] Step 4:

[0064] Server: Analyzes the received ingredient list and converts it into a format that can be passed as input data to the generation AI.

[0065] Step 5:

[0066] Server: Pass the ingredients list to the generation AI and have it generate candidate cooking recipe information.

[0067] Step 6:

[0068] Generative AI: Based on a list of ingredients, it generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[0069] Step 7:

[0070] Server: Receives the generated recipe information and provides the video generation AI with cooking instructions and a list of necessary ingredients.

[0071] Step 8:

[0072] Video generation AI: Based on recipe information, it simulates cooking steps and generates visually easy-to-understand cooking videos.

[0073] Step 9:

[0074] Video generation AI: Returns the generated video file to the server.

[0075] Step 10:

[0076] Server: Sends the recipe information and generated video together to the user's device.

[0077] Step 11:

[0078] Terminal: Analyzes the recipe information and video sent from the server and formats them for display on the user interface.

[0079] Step 12:

[0080] Device: The user interface displays recipes such as "Stir-fried chicken" and "Cabbage and carrot salad," along with cooking videos for each recipe that can be played.

[0081] Step 13:

[0082] User: Review suggested recipes and videos and select the dish they prefer.

[0083] Step 14:

[0084] User: Sees an ingredient needed for a selected recipe that is not in the refrigerator (e.g., soy sauce) and asks to purchase it within the app.

[0085] Step 15:

[0086] Terminal: Convert the list of ingredients desired for purchase into JSON format and send it to the EC site's API.

[0087] Step 16:

[0088] Server: Obtains food inventory and price information from the EC site's API and presents it to the user.

[0089] Step 17:

[0090] User: Confirms purchase.

[0091] Step 18:

[0092] Terminal: Generates a request to send purchase confirmation information to the e-commerce site's API via the server.

[0093] Step 19:

[0094] Server: Sends the received request to the e-commerce site and confirms the order.

[0095] Step 20:

[0096] E-commerce sites: process orders and initiate logistics processes quickly.

[0097] Step 21:

[0098] Server: Receives real-time delivery status information from the e-commerce site and notifies the user.

[0099] Step 22:

[0100] User: Check delivery status and wait for additional ingredients to arrive.

[0101] Step 23:

[0102] User: When additional ingredients arrive, check their arrival on the device and proceed with cooking preparations.

[0103] Step 24:

[0104] User: Starts cooking the selected dish using the provided recipe and video.

[0105] The above are the specific processing steps of the system according to the present invention. This process allows users to efficiently use ingredients in their refrigerators and enjoy a variety of dishes.

[0106] Example 1

[0107] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0108] In modern households, it is difficult to efficiently use ingredients in the refrigerator to create new dishes. Furthermore, there is a lack of convenient ways to purchase additional ingredients, which limits the variety of dishes available. Furthermore, there is a lack of ways to clearly communicate cooking steps, which can lead to users making mistakes.

[0109] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0110] In this invention, the server includes a means for inputting ingredient information, a generation AI means for analyzing the input ingredient information and generating candidate recipes, a video generation means for generating cooking procedure videos based on the generated recipes, a user interface means for displaying the generated recipes and cooking procedure videos, an e-commerce means for purchasing additional ingredients, and a means for using prompts when generating candidate recipes. This allows users to efficiently use ingredients in their refrigerators, easily create new dishes, and easily purchase additional ingredients. Furthermore, the cooking procedures can be visually understood through the videos, reducing the risk of making mistakes in cooking procedures.

[0111] The "means for inputting ingredient information" is a means by which a user inputs information about ingredients stored in the refrigerator and transmits that information to the system.

[0112] The "generative AI means" is an artificial intelligence means for analyzing input ingredient information, generating appropriate prompt sentences, and generating cooking recipes based on these.

[0113] The "video generation means" is a means for generating a video including cooking procedures based on the generated cooking recipe.

[0114] The "user interface means" is a means for displaying the generated cooking recipe and cooking procedure video to the user.

[0115] An "electronic commerce means" is a means for purchasing additional ingredients that are needed, and serves to obtain inventory and price information and assist with the purchasing process.

[0116] "Method using prompt sentences" refers to a method of generating a cooking recipe by inputting a prompt sentence created based on ingredient information into a generation AI.

[0117] This invention is a system that proposes cooking recipes using ingredients in a user's refrigerator and generates cooking videos for those recipes. This system allows users to efficiently create new cooking menus. It also allows users to easily purchase additional ingredients as needed to increase the variety of dishes.

[0118] System Overview

[0119] 1. Ingredient input method:

[0120] The user launches the application on their smartphone or PC and manually inputs the ingredients in their refrigerator. This ingredient information is converted to JSON format by the device and sent to the server.

[0121] 2. Generation AI means:

[0122] The server analyzes the received ingredient information and passes the data to the generation AI to generate a cooking recipe. Based on ingredient information such as "chicken, cabbage, and carrots," the generation AI generates multiple recipes using the following example prompt:

[0123] Prompt Sentence Examples

[0124] Please suggest a recipe that can be made using the following ingredients:

[0125] chicken meat

[0126] cabbage

[0127] Carrots

[0128] The generation AI takes the above prompt as input and generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[0129] 3. Video generation method:

[0130] The server then provides the generated recipe information to a video generation AI, which generates a video with cooking instructions in a visually easy-to-understand format. A deep learning-based video generation model, such as PaddleGAN, is used to generate the video. The generated video file is then returned to the server.

[0131] 4. User Interface Means:

[0132] The server sends the acquired recipe information and the generated video to the user's device and displays them through a user interface. The user can view the suggested recipes and videos and select their favorite dishes.

[0133] 5. Electronic commerce instruments:

[0134] If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The delivery status is notified to the user in real time.

[0135] Specific examples

[0136] A specific example will be described in which the user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator.

[0137] 1. User: Opens the smartphone app and manually enters the ingredients in the refrigerator: "chicken," "cabbage," and "carrots."

[0138] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[0139] 3. Server: Analyzes the received ingredient list, inputs "chicken, cabbage, carrots" into the generation AI, and generates the following recipe.

[0140] Stir-fried chicken

[0141] Cabbage and carrot salad

[0142] 4. Server: Requests the video generation AI to generate a cooking video for the above recipe, and returns the generated video file to the server.

[0143] 5. Server: Sends the generated recipes and videos to the user's device.

[0144] 6. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[0145] 7. User: Review suggested recipes and videos and select "Chicken Stir-fry."

[0146] 8. E-commerce method: If the user does not have the necessary additional ingredients (e.g., soy sauce), the user requests to purchase the additional ingredients through the app. The device sends the list of ingredients to the e-commerce site.

[0147] 9. Server: Retrieves inventory and price information from the e-commerce site's API and presents it to the user.

[0148] 10. User: Confirms purchase.

[0149] 11. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[0150] 12. Server: Notifies the user of delivery status in real time.

[0151] 13. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[0152] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, reduces food waste, and improves the user experience.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Program processing flow

[0155] Step 1: Enter ingredient information

[0156] User: Launches the application on their smartphone or PC and manually inputs the ingredients they have in their refrigerator. They add ingredients such as "chicken," "cabbage," and "carrots" to the input fields and clicks the submit button.

[0157] Input: Information about ingredients in the refrigerator (e.g., "chicken," "cabbage," "carrots")

[0158] Terminal: Convert the input ingredient information into JSON format.

[0159] Output: JSON formatted ingredient information

[0160] json

[0161] {

[0162] "ingredients": ["chicken", "cabbage", "carrot"]

[0163] }

[0164] What happens: The device collects user-entered information and executes code to convert it into a JSON object, for example, using JavaScript or Python program code.

[0165] Step 2: Submit ingredient information

[0166] Terminal: Sends the converted JSON formatted ingredient information to the server.

[0167] Input: JSON formatted ingredient information

[0168] Output: Ingredient information sent to the server

[0169] What it does: Executes code to create an HTTP POST request and send it to the specified URL on the server.

[0170] Step 3: Recipe generation

[0171] Server: Analyze the received JSON formatted ingredient information.

[0172] Input: JSON formatted ingredient information

[0173] Output: A list of ingredients

[0174] What it does: Uses a JSON parsing library to get the list of ingredients.

[0175] Server: Enters prompt text into the generation AI based on the analyzed data.

[0176] Input: List of ingredients

[0177] Generative AI: Generates cooking recipes based on prompts.

[0178] Prompt Sentence Examples

[0179] Please suggest a recipe that can be made using the following ingredients:

[0180] chicken meat

[0181] cabbage

[0182] Carrots

[0183] Output: Multiple cooking recipes (e.g. "Stir-fried chicken" and "Cabbage and carrot salad")

[0184] Specific operation: Send a prompt to the generation AI (e.g., GPT model) and obtain the generated recipe list.

[0185] Step 4: Save the recipe information

[0186] Server: Stores the generated recipe information in a database.

[0187] Input: Multiple cooking recipes

[0188] Output: Recipe information stored in the database

[0189] What it does: Executes code to add recipe information to the database using an SQL INSERT statement.

[0190] Step 5: Generate cooking videos

[0191] Server: Provides recipe information to the video generation AI.

[0192] Input: Cooking recipe

[0193] Output: Cooking videos generated by video generation AI

[0194] Specific operation: Create an HTTP POST request and send recipe data to the video generation AI API.

[0195] Video generation AI: Generates videos including cooking instructions based on the provided recipe information.

[0196] Input: Cooking recipe

[0197] Output: Cooking videos

[0198] What it does: Runs code that uses a deep learning model to generate a series of image frames and stitches them together into a video.

[0199] Step 6: Submit your recipe and video

[0200] Server: Sends the generated recipe information and cooking videos to the user's device.

[0201] Input: Cooking recipes, cooking videos

[0202] Output: Recipe information and video sent to the user's device

[0203] Specific behavior: Executes code to send data to the user's device via an HTTP GET or POST request.

[0204] Step 7: Display in the user interface

[0205] Terminal: Parses the received data and displays it in the application's user interface.

[0206] Input: Cooking recipes, cooking videos

[0207] Output: Recipe information and video displayed in the user interface

[0208] Specific operation: Analyzes the received JSON data, displays the recipe title, ingredients, and cooking steps in text format, and simultaneously executes code to play the cooking video on a video player.

[0209] Step 8: Purchase additional ingredients

[0210] User: If an additional ingredient (e.g. soy sauce) is needed, the user wishes to purchase the additional ingredient in the app. Click the purchase button and select the required ingredient.

[0211] Terminal: Send the list of ingredients you wish to purchase in JSON format to the e-commerce site.

[0212] Input: List of ingredients you wish to purchase

[0213] Output: Ingredient list sent to the e-commerce site

[0214] What it does: Executes code to create an HTTP POST request and send it to the e-commerce site's API.

[0215] Step 9: Get inventory and pricing information

[0216] Server: Obtains inventory and price information from the e-commerce site's API.

[0217] Input: List of ingredients you wish to purchase

[0218] Output: Stock and price information obtained from the e-commerce site

[0219] Specific behavior: Executes code to retrieve information from the e-commerce site's API via an HTTP GET request and receive it in JSON format.

[0220] Server: Presents stock and price information to the user. Sends data to the device and displays it on the user's screen.

[0221] Input: Stock and price information obtained from the e-commerce site

[0222] Output: Stock and pricing information displayed to the user.

[0223] What it does: Sends data to the device and executes code to display it in the user interface.

[0224] Step 10: Confirm purchase and process order

[0225] User: Confirms purchase by clicking the Confirm button on the screen.

[0226] Input: Purchase confirmation instructions

[0227] Output: Confirmed information

[0228] What it does: Receives input from the user interface and executes code to send purchase confirmation information to the server.

[0229] Server: Receives purchase confirmation information, sends it to the EC site's API, confirms the order, and begins the shipping process.

[0230] Input: Purchase confirmation information

[0231] Output: Order information for the e-commerce site

[0232] What it does: Executes code to create an HTTP POST request and send it to the e-commerce site's API.

[0233] Step 11: Delivery Status Notification

[0234] Server: Notify the user of the delivery status in real time, for example via push notifications or in-app messages.

[0235] Input: Delivery status information

[0236] Output: User notification

[0237] What it does: Executes code to notify the user of delivery status using a push notification server or an in-app notification system.

[0238] As described above, this system efficiently utilizes ingredients in the user's refrigerator and provides new cooking recipes and cooking videos, thereby increasing the variety of dishes available and simplifying the purchase of additional ingredients.

[0239] (Application example 1)

[0240] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0241] Conventional recipe suggestion systems offer limited options for efficiently preparing dishes using ingredients users already have, and lack content that allows users to visually understand cooking methods. This makes it time-consuming and labor-intensive for users to create new recipes, making it difficult to efficiently utilize ingredients. Purchasing additional ingredients is also time-consuming and often burdensome for users. This limits the variety of dishes available and leads to problems such as wasted ingredients.

[0242] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0243] In this invention, the server includes a means for inputting food information, a generative AI model means for analyzing the input food information and generating candidate recipes, a video generation means for generating cooking procedure videos based on the generated recipes, a user interface means for displaying the generated recipes and cooking procedure videos, and an online commerce means for purchasing additional ingredients. This allows users to efficiently use ingredients in their refrigerators and easily create new recipes. Furthermore, by making cooking procedures easier to understand visually through videos, cooking time is reduced and additional ingredients can be purchased more smoothly. This reduces food waste and improves the user experience.

[0244] The "means for inputting food information" is an interface that allows a user to input information about ingredients and other food items in the refrigerator using a terminal that the user owns and transmits this information to the system.

[0245] The "generative AI model means" is a module that uses artificial intelligence technology to generate optimal cooking methods based on input food information.

[0246] The "video generation means" is a technology for generating a visual video of cooking procedures based on the generated recipes, so that users can easily understand how to cook.

[0247] The "user interface means" is an interface that displays the generated recipe and cooking procedure videos to the user and allows the user to view and operate them.

[0248] "Online commerce means" refers to a means by which a user can make an online transaction to purchase additional ingredients that the user needs.

[0249] This system proposes recipes using ingredients that the user has on hand, and generates and distributes videos of the cooking steps, allowing users to efficiently create new recipes and easily purchase any additional ingredients they need.

[0250] System Overview

[0251] The system includes the following means:

[0252] 1. Means for entering food information:

[0253] The user inputs the ingredients in the refrigerator using a device such as a smartphone. This ingredient information is converted into JSON format by the device and sent to the server.

[0254] 2. Generative AI model means:

[0255] The server analyzes the received ingredient information and passes the data to a generative AI to generate recipes. For example, based on ingredient information such as "chicken, cabbage, and carrots," the generative AI model generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[0256] 3. Video generation method:

[0257] The server provides the generated recipe information to the video generation AI, which generates a video with cooking instructions in a visually easy-to-understand format. The generated video file is then returned to the server.

[0258] 4. User Interface Means:

[0259] The server sends the acquired recipe information and the generated video to the user's device and displays them through a user interface. The user can view the suggested recipes and videos and select their favorite dishes.

[0260] 5. Online commerce methods:

[0261] If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The delivery status is notified to the user in real time.

[0262] Program processing

[0263] The system uses the following hardware and software:

[0264] Hardware

[0265] Smartphones and tablets (user devices)

[0266] Cloud Server

[0267] software

[0268] On the server side, recipe generation AI (e.g., GPT-4) and video generation AI (e.g., DALL-E and Deep Video) are used.

[0269] On the client side, a communication library (e.g., requests) is used to send and receive data in JSON format.

[0270] A standard web front-end framework (e.g., React, Angular, etc.) is used to display the user interface.

[0271] Specific examples

[0272] As a specific example, consider the case where the user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator.

[0273] Prompt Sentence Examples

[0274] Suggest a simple and delicious recipe using the following ingredients you have in your fridge:

[0275] chicken meat

[0276] cabbage

[0277] Carrots

[0278] The server passes this prompt to the generation AI, which generates a recipe like this:

[0279] "Stir-fried chicken" and "Cabbage and carrot salad"

[0280] The server then passes this recipe information to a video generation AI, which generates a video of the cooking steps. The generated data is stored in cloud data storage and sent to the user's device. Users can view these recipes and videos through a smartphone app and begin cooking based on the recipe they select. If additional ingredients are required, they can easily purchase them through the e-commerce site.

[0281] This system allows users to make the most of the ingredients they have, reduce waste, and easily enjoy new cooking menus.

[0282] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0283] Step 1:

[0284] The user launches the app on their smartphone and inputs the ingredients they have in their refrigerator. The user enters the specific ingredients they have (e.g., chicken, cabbage, carrots) into the app. This information is converted into a data format (e.g., JSON) and sent to the server.

[0285] Input: Ingredient information entered by the user

[0286] Output: Ingredient information in JSON format

[0287] Step 2:

[0288] The server analyzes the received ingredient information and passes the data to the generative AI model. The server then generates a prompt based on the ingredient information and inputs it into the generative AI model (e.g., GPT-4). During this analysis, the input ingredient information is formatted appropriately.

[0289] Input: JSON formatted ingredient information

[0290] Output: A prompt to be passed to the generative AI model

[0291] Step 3:

[0292] The server uses a generative AI model to generate multiple recipes based on the input prompt, which analyzes and generates several candidate recipes based on the user's ingredients.

[0293] Input: prompt statement

[0294] Output: Multiple generated recipes

[0295] Step 4:

[0296] The server provides the generated recipe information to the video generation AI, which then generates cooking instructions as videos. The video generation AI then generates video files that visually explain the steps of each recipe in an easy-to-understand format.

[0297] Input: Generated recipe information

[0298] Output: Video file visualizing the cooking steps

[0299] Step 5:

[0300] The server sends the generated recipes and videos to the user's device, stores them in cloud storage, and delivers them to the user's app in real time.

[0301] Input: Generated recipes and videos

[0302] Output: User device where recipes and videos are delivered

[0303] Step 6:

[0304] Users can browse suggested recipes and videos through the app and select their preferred dish. Users can then select one of the multiple options on the app and see how to cook it.

[0305] Input: Distributed recipes and videos

[0306] Output: User selected recipe

[0307] Step 7:

[0308] If the user wishes to purchase additional ingredients, the device sends a list of the ingredients they wish to purchase to the EC site. The device then creates a list of the ingredients needed based on the selected recipe and sends the data to the EC site.

[0309] Input: A list of additional ingredients based on the recipe selected by the user

[0310] Output: Ingredient list sent to the e-commerce site

[0311] Step 8:

[0312] The server calls the API of the e-commerce site, obtains stock and price information for each ingredient, and presents it to the user. The server analyzes the API response and presents the information in a format that is easy for the user to understand.

[0313] Input: List of additional ingredients sent

[0314] Output: Retrieved inventory and price information

[0315] Step 9:

[0316] The user confirms the purchase procedure through the app and confirms the purchase. The app receives the user's input and sends it to the server.

[0317] Input: User's purchase confirmation information

[0318] Output: Purchase confirmation information

[0319] Step 10:

[0320] The server sends the purchase confirmation information to the e-commerce site's API to complete the order process. The server also sends the order information via the API to start the logistics process.

[0321] Input: User's purchase confirmation information

[0322] Output: Order information sent to the e-commerce site

[0323] Step 11:

[0324] The server notifies the user of the delivery status in real time. The server obtains the delivery status from the EC site and notifies the user.

[0325] Input: Delivery status from the e-commerce site

[0326] Output: Real-time notification of delivery status

[0327] This process allows users to effectively use the ingredients they have and smoothly purchase any additional ingredients they need. The generated cooking video also reduces the time and effort required for cooking.

[0328] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0329] The present invention is a system that suggests cooking recipes using ingredients in the user's refrigerator and generates cooking videos for those recipes. By further combining this system with an emotion engine that recognizes the user's emotions, it is possible to suggest recipes and videos that correspond to the user's emotional state. This system not only enables users to efficiently create new cooking menus, but also provides a more satisfying experience by receiving suggestions that match their emotions. Specific program processing for this system is described below.

[0330] System Overview

[0331] 1. Ingredient input method: The user launches the application on their smartphone or PC and inputs the ingredients they have in their refrigerator. This ingredient information is converted to JSON format by the device and sent to the server.

[0332] 2. Recipe generation: The server analyzes the received ingredient information and passes the data to the generation AI to generate a cooking recipe. For example, based on ingredient information such as "chicken, cabbage, and carrots," it generates recipes such as "stir-fried chicken" or "cabbage and carrot salad."

[0333] 3. Emotion Engine: When a user is using the application, sensors such as a camera and microphone are used to analyze the user's facial expressions and voice to recognize their emotional state. This emotional information is then sent to the server.

[0334] 4. Emotion-based suggestions: The server analyzes the emotion information received from the emotion engine and adjusts the recipe suggestions based on the user's current emotional state. For example, if the user is feeling stressed, it will suggest dishes with a relaxing effect.

[0335] 5. Video generation method: The server provides recipe information and emotion information to the video generation AI, which generates a video with cooking instructions that match the emotion and is easy to understand visually. The generated video file is returned to the server.

[0336] 6. User interface means: The server sends the acquired recipe information, emotion information, and generated video to the user's device and displays them through the user interface. The user can view the suggested recipes and videos and choose their favorite dishes.

[0337] 7. E-commerce method: If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The delivery status is notified to the user in real time.

[0338] Specific examples

[0339] As a specific example, consider a case where a user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator, and the emotion engine recognizes that the user is feeling stressed.

[0340] 1. User: Opens the app on his smartphone and enters the ingredients in his refrigerator: "chicken," "cabbage," and "carrots."

[0341] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[0342] 3. Server: Analyzes the received ingredient list, inputs "chicken, cabbage, carrots" into the generation AI, and generates the following recipes ("stir-fried chicken" and "cabbage and carrot salad").

[0343] 4. Emotion engine: Analyzes the user's facial expressions and voice and notifies the server that the user is feeling stressed.

[0344] 5. Server: Analyzes information from the emotion engine and prioritizes recipes that have a relaxing effect.

[0345] 6. Video generation method: The video generation AI is requested to generate a cooking video for the "cabbage and carrot salad" recipe, which is particularly relaxing from the above recipes. The generated video file is returned to the server.

[0346] 7. Server: Sends the generated recipes and videos to the user's device.

[0347] 8. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[0348] 9. User: Review suggested relaxing recipes and videos and choose to cook.

[0349] 10. E-commerce method: If the user does not have the necessary additional ingredients (e.g., olive oil), the user requests to purchase the additional ingredients through the app. The device sends the list of ingredients to the e-commerce site.

[0350] 11. Server: Retrieves olive oil inventory and price information from the e-commerce site's API and presents it to the user.

[0351] 12. User: Confirms purchase.

[0352] 13. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[0353] 14. E-commerce: Process orders and initiate logistics processes in a short time.

[0354] 15. Server: Notifies the user of delivery status in real time.

[0355] 16. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[0356] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, and provides a more satisfying cooking experience by making suggestions based on the user's emotional state.

[0357] The processing flow will be explained below.

[0358] Step 1:

[0359] User: Launches the application on their smartphone or PC and accesses the screen where they can input the ingredients in their refrigerator.

[0360] Step 2:

[0361] User: Enter the ingredients they currently have on hand, such as "chicken," "cabbage," and "carrots," into the text field.

[0362] Step 3:

[0363] Terminal: Convert the input ingredient list into JSON format and send it to the server.

[0364] Step 4:

[0365] Server: Analyzes the received ingredient list and converts it into a format that can be passed as input data to the generation AI.

[0366] Step 5:

[0367] Server: Pass the ingredients list to the generation AI and have it generate candidate cooking recipe information.

[0368] Step 6:

[0369] Generative AI: Based on a list of ingredients, it generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[0370] Step 7:

[0371] Server: Receives the generated recipe information.

[0372] Step 8:

[0373] Emotion engine: When a user is using an application, the system captures the user's facial expressions and voice data via the camera and microphone, and analyzes this to recognize the user's emotional state.

[0374] Step 9:

[0375] Emotion engine: Sends the recognized emotional state to the server. For example, if the user is feeling stressed, it sends the data of the stress state.

[0376] Step 10:

[0377] Server: Analyzes the emotional information received from the emotion engine and adjusts recipe suggestions based on the user's current emotional state, for example, prioritizing dishes with a relaxing effect.

[0378] Step 11:

[0379] Server: Provides recipe information and emotional information to the video generation AI and asks it to generate a cooking instruction video that matches the emotion.

[0380] Step 12:

[0381] Video generation AI: Based on recipe information, it simulates cooking steps and generates visually easy-to-understand cooking videos.

[0382] Step 13:

[0383] Video generation AI: Returns the generated video file to the server.

[0384] Step 14:

[0385] Server: Sends the recipe information and generated video together to the user's device.

[0386] Step 15:

[0387] Terminal: Analyzes the recipe information and video sent from the server and formats them for display on the user interface.

[0388] Step 16:

[0389] Device: The user interface displays recipes such as "Stir-fried chicken" and "Cabbage and carrot salad," along with cooking videos for each recipe that can be played.

[0390] Step 17:

[0391] User: Review suggested recipes and videos and select the dish they prefer.

[0392] Step 18:

[0393] User: Sees an ingredient needed for a selected recipe that is not in the refrigerator (e.g., soy sauce) and asks to purchase it within the app.

[0394] Step 19:

[0395] Terminal: Convert the list of ingredients desired for purchase into JSON format and send it to the EC site's API.

[0396] Step 20:

[0397] Server: Obtains food inventory and price information from the EC site's API and presents it to the user.

[0398] Step 21:

[0399] User: Confirms purchase.

[0400] Step 22:

[0401] Terminal: Generates a request to send purchase confirmation information to the e-commerce site's API via the server.

[0402] Step 23:

[0403] Server: Sends the received request to the e-commerce site and confirms the order.

[0404] Step 24:

[0405] E-commerce sites: process orders and initiate logistics processes quickly.

[0406] Step 25:

[0407] Server: Receives real-time delivery status information from the e-commerce site and notifies the user.

[0408] Step 26:

[0409] User: Check delivery status and wait for additional ingredients to arrive.

[0410] Step 27:

[0411] User: When additional ingredients arrive, check their arrival on the device and proceed with cooking preparations.

[0412] Step 28:

[0413] User: Starts cooking the selected dish using the provided recipe and video.

[0414] The above are the specific processing steps of the system according to the present invention. This process not only allows users to efficiently use the ingredients in their refrigerator to enjoy a variety of dishes, but also provides a more satisfying cooking experience by receiving recipe suggestions that match their emotions.

[0415] Example 2

[0416] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0417] Conventional recipe suggestion systems are unable to consider the user's emotional state when proposing recipes, making it difficult to enhance the user's psychological satisfaction. Furthermore, they lack a means to provide easy-to-understand visual instructions for cooking, making them difficult to use, especially for beginners. Another issue is the complicated process of purchasing additional ingredients, which makes it difficult to start cooking smoothly.

[0418] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0419] In this invention, the server includes a means for inputting ingredient information, an artificial intelligence means for analyzing the input ingredient information and generating candidate recipes, an emotion engine means for recognizing the user's emotional state and adjusting the cooking recipe based on the recognized emotional information, a means for generating cooking procedure videos based on the generated cooking recipe, a user interface means for displaying the generated cooking recipe and cooking procedure videos, and an electronic commerce means for purchasing any additional ingredients required. This allows the user to receive recipe suggestions suited to their emotional state and to proceed with cooking while referring to the visually easy-to-understand cooking videos. It also makes it easy to purchase any additional ingredients required.

[0420] The "means for inputting ingredient information" is an interface that allows users to input ingredients in the refrigerator as text or images via a smartphone or PC application.

[0421] The "artificial intelligence means" is a system that analyzes input ingredient information and generates candidate cooking recipes using a generative AI model.

[0422] The "emotion engine means" is a function that analyzes the user's facial expressions and voice, recognizes the user's emotional state, and adjusts the cooking recipe based on the emotional information.

[0423] The "means for generating cooking procedure videos" is a technology for generating cooking procedure videos that are visually easy to understand based on the generated cooking recipe.

[0424] The "user interface means" is an interface for displaying the generated cooking recipe and cooking procedure video to the user.

[0425] "Electronic commerce means" refers to a system that provides online shopping functionality for purchasing additional ingredients needed.

[0426] The present invention is a system that suggests cooking recipes using ingredients in the user's refrigerator and generates cooking videos for those recipes. Furthermore, by combining it with an emotion engine that recognizes the user's emotional state, it suggests recipes and videos according to that emotional state. This system allows users to efficiently create new cooking menus, and by receiving suggestions that match their emotions, it increases their satisfaction. The system's program processing is explained in detail below.

[0427] System Overview

[0428] Entering ingredient information

[0429] Users enter information about ingredients in their refrigerators through a smartphone or PC application. Input methods include text entry and uploading images taken with a camera. The entered information is converted to JSON format by the device and sent to the server.

[0430] Recipe Generation

[0431] The server analyzes the received JSON-formatted ingredient information and sends a prompt to the generative AI model (e.g., GPT-3) saying, "Please suggest a cooking recipe that includes chicken, cabbage, and carrots." The generative AI model generates multiple recipes based on the received prompt.

[0432] emotion recognition

[0433] When a user is using an application, sensors such as a camera and microphone are used to record facial expressions and voice. The device then uses the recorded data to analyze the user's emotional state using an emotion recognition engine (e.g., Microsoft Azure Emotion API). The analyzed emotional information is converted into JSON format and sent to the server.

[0434] Emotion-based recipe suggestions

[0435] The server analyzes the received emotional information and adjusts the recipe suggestions based on the user's current emotional state. For example, if the user is feeling stressed, it will prioritize suggestions of cooking recipes that have a relaxing effect. As a specific example, it generates "stir-fried chicken" and "cabbage and carrot salad," and prioritizes displaying the "cabbage and carrot salad," which has the effect of relieving stress.

[0436] Cooking video generation

[0437] The server requests a video generation AI (e.g., Synthesia) to generate a video explaining the cooking steps for the specified recipe based on the emotional state and recipe information. The generated video file is sent back to the server and provided to the user.

[0438] User display and operation

[0439] The server sends the acquired recipe information, emotion information, and generated cooking videos to the user's device, which analyzes the information and displays it through a user interface. The user can then check and operate the suggested recipes and videos.

[0440] Purchase additional ingredients

[0441] If the user does not have the necessary ingredients to prepare a suggested recipe, they can request to purchase the ingredients through the app. The device sends a list of the ingredients they wish to purchase to the EC site via an e-commerce transaction method. The server retrieves inventory and price information from the EC site's API and presents it to the user. Once the user confirms the purchase, the server sends the order information to the EC site's API and confirms the order. The user is notified of the delivery status in real time.

[0442] Specific examples

[0443] As a specific example, consider a case where a user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator, and the emotion engine recognizes that the user is feeling stressed.

[0444] 1. User: Opens the app on his smartphone and enters the ingredients in his refrigerator: "chicken," "cabbage," and "carrots."

[0445] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[0446] 3. Server: Analyzes the received list of ingredients, inputs "chicken, cabbage, carrots" into a generation AI (e.g., GPT-3), and generates recipes such as the following ("stir-fried chicken" and "cabbage and carrot salad").

[0447] 4. Emotion engine: Analyzes the user's facial expressions and voice and notifies the server that the user is feeling stressed.

[0448] 5. Server: Analyzes information from the emotion engine and prioritizes recipes that have a relaxing effect.

[0449] 6. Video generation method: A video generation AI (e.g., Synthesia) is requested to generate a cooking video for the "cabbage and carrot salad" recipe, which is particularly relaxing from the above recipes. The generated video file is returned to the server.

[0450] 7. Server: Sends the generated recipes and videos to the user's device.

[0451] 8. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[0452] 9. User: Review suggested relaxing recipes and videos and choose to cook.

[0453] 10. E-commerce method: If you don't have the additional ingredients you need (e.g. olive oil), submit a list of ingredients you want to purchase to the e-commerce site.

[0454] 11. Server: Retrieves olive oil inventory and price information from the e-commerce site's API and presents it to the user.

[0455] 12. User: Confirms purchase.

[0456] 13. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[0457] 14. E-commerce website: Processes orders and initiates the delivery process.

[0458] 15. Server: Notifies the user of delivery status in real time.

[0459] 16. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[0460] Prompt Sentence Examples

[0461] An example of a prompt sentence to input into the generative AI model is, "Please suggest a relaxing cooking recipe that includes chicken, cabbage, and carrots."

[0462] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, and provides a more satisfying cooking experience by making suggestions based on the user's emotional state.

[0463] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0464] Step 1:

[0465] The user enters ingredient information

[0466] Users launch the application on their smartphone or PC and enter information about the ingredients in their refrigerator. Specifically, they enter "chicken," "cabbage," and "carrots" in the text input field and press the send button. Alternatively, they can take a photo of the ingredients and upload it.

[0467] input:

[0468] Ingredient information (text or image)

[0469] output:

[0470] Ingredient information is saved on the device

[0471] Step 2:

[0472] The device analyzes the ingredient information and sends it to the server.

[0473] The device extracts the names of ingredients from the input text data as is, or from image data using an image recognition engine (e.g., Google Cloud Vision API). The extracted names are converted to JSON format and sent to the server.

[0474] input:

[0475] Ingredient information (text or image)

[0476] Data processing / calculation:

[0477] Convert text information (directly into JSON)

[0478] Recognition processing using an image analysis engine (image → text information)

[0479] output:

[0480] JSON formatted ingredient information

[0481] Step 3:

[0482] The server analyzes the sent ingredient information and sends a prompt to the generative AI model.

[0483] The server analyzes the received JSON-formatted ingredient information and sends a prompt to the generative AI model (e.g., GPT-3) such as, "Please suggest a cooking recipe that includes chicken, cabbage, and carrots."

[0484] input:

[0485] JSON formatted ingredient information

[0486] Data processing / calculation:

[0487] Prompt statement generation and sending (creating a prompt statement and sending it)

[0488] output:

[0489] Prompt to send to the generative AI model

[0490] Step 4:

[0491] A generative AI model generates cooking recipes based on prompts

[0492] The generative AI model receives a prompt and generates potential recipes based on the prompt, such as "stir-fried chicken" or "cabbage and carrot salad."

[0493] input:

[0494] Prompt statement

[0495] Data Calculation:

[0496] Recipe generation using natural language processing

[0497] output:

[0498] Cooking recipe information

[0499] Step 5:

[0500] The server receives the generated recipe and obtains the emotion information.

[0501] The server stores the generated recipe information and then prepares to acquire the user's emotional information, specifically, issues an instruction to the terminal to acquire the user's current emotional state.

[0502] input:

[0503] Cooking recipe information

[0504] output:

[0505] Instructions for obtaining emotional information

[0506] Step 6:

[0507] The device records the user's facial expressions and voice and analyzes them using an emotion engine.

[0508] When a user is using an application, the device's camera and microphone are used to record facial expressions and voice. The device then uses the recorded data to analyze the user's emotional state using an emotion recognition engine (e.g., Microsoft Azure Emotion API).

[0509] input:

[0510] Facial expression data, voice data

[0511] Data processing / calculation:

[0512] Analysis by emotion recognition engine

[0513] output:

[0514] Emotion information in JSON format

[0515] Step 7:

[0516] The device sends the analyzed emotional information to the server.

[0517] The device converts the analysis results into emotional information in JSON format and sends it to the server. For example, information such as "Emotion: Stress" is sent.

[0518] input:

[0519] Emotion information (JSON format)

[0520] output:

[0521] Send emotion information to the server

[0522] Step 8:

[0523] The server analyzes the emotional information and adjusts the recipe suggestions.

[0524] The server analyzes the received emotional information and adjusts recipe suggestions based on the user's current emotional state. For example, if a user is feeling stressed, it will prioritize recipes that have a relaxing effect.

[0525] input:

[0526] Emotional information, cooking recipe information

[0527] Data processing / calculation:

[0528] Recipe adjustment based on emotion information

[0529] output:

[0530] Adjusted cooking recipe information

[0531] Step 9:

[0532] The server requests the video generation AI to generate a cooking procedure video.

[0533] Based on the emotional state and recipe information, the server requests a video generation AI (e.g., Synthesia) to generate a video explaining the cooking steps for the specified recipe.

[0534] input:

[0535] Adjusted cooking recipe information

[0536] Data Calculation:

[0537] Video generation by video generation AI

[0538] output:

[0539] Cooking procedure video

[0540] Step 10:

[0541] The server sends the generated recipe and video to the device.

[0542] The server sends the generated recipe information and cooking procedure video to the user's terminal.

[0543] input:

[0544] Cooking procedure videos, adjusted cooking recipe information

[0545] output:

[0546] Send recipe information and videos to your device

[0547] Step 11:

[0548] The device analyzes the received information and displays it on the user interface.

[0549] The device analyzes the recipe information and cooking video sent from the server and displays them on the user interface. The user can then check the suggested recipes and videos and choose their preferred dish.

[0550] input:

[0551] Recipe information, cooking procedure videos

[0552] output:

[0553] Display in the user interface

[0554] Step 12:

[0555] The user performs an operation to purchase any additional ingredients needed.

[0556] If the user does not have the necessary ingredients to prepare the suggested recipe, they can request to purchase the ingredients through the app. They can create a list of ingredients they wish to purchase and send it to their device.

[0557] input:

[0558] Missing ingredient information

[0559] output:

[0560] List of ingredients you want to buy

[0561] Step 13:

[0562] The device sends the list of ingredients desired to be purchased to the e-commerce site and displays the information.

[0563] The terminal sends the list of ingredients desired to be purchased to the EC site via the electronic commerce means. The server retrieves inventory and price information from the EC site's API and presents it to the user.

[0564] input:

[0565] List of ingredients you want to buy

[0566] output:

[0567] Send to EC site and display stock and price information

[0568] Step 14:

[0569] The user confirms the purchase and the server sends the order information to the e-commerce site.

[0570] The user checks the inventory and price information provided and confirms the purchase. The server sends the purchase confirmation information to the EC site's API and confirms the order. The EC site processes the order and starts the delivery process.

[0571] input:

[0572] Purchase confirmation information

[0573] output:

[0574] Send order information to the EC site

[0575] Step 15:

[0576] The server notifies the user of the delivery status in real time.

[0577] The server notifies the user of the delivery status in real time. When the additional ingredients arrive, the user can check the arrival and start cooking. The user can cook according to the recipe while watching the provided video.

[0578] input:

[0579] Delivery Status Information

[0580] output:

[0581] Notify users of delivery status

[0582] (Application example 2)

[0583] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0584] While conventional cooking recommendation systems can suggest recipes based on the ingredients in the refrigerator, they are unable to suggest optimal dishes based on the user's emotional state. This makes it difficult to accurately suggest dishes desired when the user is in a specific emotional state, and there are a lack of means to improve the user experience. Furthermore, there is a lack of functionality to immediately order the suggested dishes and track their delivery status.

[0585] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting ingredient information, artificial intelligence means for analyzing the input ingredient information and generating candidate recipes, means for generating cooking procedure videos based on the generated recipes, user interface means for displaying the generated recipes and cooking procedure videos, e-commerce means for purchasing additional ingredients, an emotion engine for recognizing the user's emotions, means for suggesting recipes based on the user's emotional state, means for ordering the suggested dishes from an external service, and means for tracking the delivery status of the ordered dishes. This enables the server to suggest optimal dishes based on the user's emotional state, and to quickly order the suggested dishes and track their delivery status.

[0586] The "means for inputting ingredient information" refers to the interface or terminal function that allows the user to input ingredient information about ingredients stored in the refrigerator.

[0587] The "artificial intelligence means for analyzing input ingredient information and generating candidate cooking recipes" refers to an artificial intelligence algorithm and calculation mechanism for generating appropriate cooking recipes based on the ingredient information input by the user.

[0588] "Means for generating videos of cooking steps based on generated cooking recipes" refers to technology and devices for creating videos of cooking steps that are visually easy to understand based on cooking recipes generated by AI.

[0589] The "user interface means for displaying the generated cooking recipe and cooking procedure video" refers to an interface and device that allows the user to visually check the generated cooking recipe and cooking procedure video.

[0590] "Electronic commerce means for purchasing additional ingredients" refers to a system and procedure for purchasing ingredients that are in short supply via the Internet.

[0591] An "emotion engine that recognizes user emotions" is a combination of software and hardware that uses a camera and microphone to analyze emotions from the user's facial expressions and voice and recognize those emotions.

[0592] The "means for suggesting recipes according to the user's emotional state" refers to an algorithm and means for selecting and suggesting optimal cooking recipes based on the user's recognized emotions.

[0593] "Means for ordering suggested dishes from an external service" refers to a system and function for ordering the dishes selected by the user from an external food delivery service or the like.

[0594] The "means for tracking the delivery status of an ordered meal" refers to a system and technology for tracking the current delivery status of an ordered meal in real time and notifying the user.

[0595] The present invention is a system that suggests optimal dishes based on a user's emotional state and orders them through an external food delivery service. This system allows users to easily order dishes that match their emotions and track delivery status in real time. Specific embodiments of this system are described below.

[0596] System Overview

[0597] The system of the present invention comprises the following components:

[0598] 1. Ingredient input method:

[0599] Users can input information about ingredients in their refrigerators using an application on their smartphone or PC. Similar applications include smart fridge management apps.

[0600] 2. Artificial Intelligence Means:

[0601] The system analyzes the input ingredient information and generates candidate recipes. This analysis is performed using a generative AI model. For example, if "chicken, cabbage, carrots" is input, recipes such as "stir-fried chicken" and "cabbage and carrot salad" will be generated.

[0602] 3. Video generation method:

[0603] Based on the generated recipe, a video showing the cooking steps is generated. This video generation is also done using an AI model. An intuitive and easy-to-understand video is created for each recipe.

[0604] 4. User Interface Means:

[0605] The generated recipes and cooking video are displayed on the user's device interface, allowing the user to select which dish to cook based on this information.

[0606] 5. Electronic commerce instruments:

[0607] It provides a means for users to purchase additional ingredients they need. When users input the ingredients they are missing, it connects with the API of the e-commerce site to carry out the online purchase process based on that information.

[0608] 6. Emotion Engine:

[0609] The system analyzes the user's facial expressions and voice to recognize their emotional state. This analysis is performed using a camera and microphone, and emotion analysis software such as EmotionEngine is used.

[0610] 7. Emotion-based recipe suggestion method:

[0611] Based on the emotional information obtained from the emotion engine, the system suggests recipes that are optimal for the user's current emotional state. For example, if the user is feeling stressed, the system suggests dishes that have a relaxing effect.

[0612] 8. How to order from third-party services:

[0613] Order the suggested dishes from an external food delivery service, specifically by placing an order with the restaurant via an API such as FoodDeliveryService.

[0614] 9. Delivery status tracking:

[0615] Track your order delivery in real time and get notified every time the delivery status changes.

[0616] Program processing overview

[0617] The server includes a means for inputting ingredient information, an artificial intelligence means for analyzing the input ingredient information and generating candidate recipes, and a means for generating videos of cooking procedures based on the generated recipes, thereby enabling the server to suggest optimal dishes according to the user's emotional state, and to quickly order the dishes and track their delivery status.

[0618] Specific examples

[0619] For example, if a user inputs that they have "chicken," "cabbage," and "carrots" in their refrigerator, and the app analyzes their facial expressions and voice and determines that they are feeling stressed, it will suggest a relaxing "cabbage and carrot salad." When the user selects that recipe, it places an order through an external food delivery service and tracks the progress of the ingredients in real time until they are delivered.

[0620] Prompt Sentence Examples

[0621] "A system that suggests the best meal for a stressed user who has chicken, cabbage, and carrots in the refrigerator, orders the meal, and tracks its delivery."

[0622] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0623] Step 1:

[0624] The user launches the application on their smartphone or PC and enters information about ingredients in the refrigerator. The ingredient information entered by the user is converted to JSON format data by the device and sent to the server. This generates (input: ingredient information entered by the user) (output: ingredient data in JSON format).

[0625] Step 2:

[0626] The server analyzes the received JSON-formatted ingredient information and generates a cooking recipe based on the ingredient information. This process uses a generative AI model. For example, if the data "chicken, cabbage, carrots" is input, the server generates candidate recipes such as "stir-fried chicken" or "cabbage and carrot salad." (Input: JSON-formatted ingredient data) (Output: candidate cooking recipe) is generated.

[0627] Step 3:

[0628] When a user is operating an application, the device's camera and microphone are used to detect the user's facial expressions and voice, which are then analyzed by the emotion engine. The analysis results (e.g., information on the user's stress level) are sent to the server, where (input: user's facial expression data and voice data) (output: emotion analysis results) are generated.

[0629] Step 4:

[0630] The server uses the emotion analysis results received from the emotion engine to prioritize recipe suggestions that match the user's emotional state. In this case, recipes that are expected to have a relaxing effect or improve mood are selected. (Input: Candidate recipes and emotion analysis results) (Output: Optimal recipe suggestions based on emotions) are generated.

[0631] Step 5:

[0632] The server generates a cooking procedure video based on the generated recipe using a video generation means. The generated cooking procedure video is returned to the server. (Input: optimal cooking recipe) (Output: cooking procedure video) is generated.

[0633] Step 6:

[0634] The server sends the generated recipes and cooking instruction videos to the user's device. The user can check these suggestions in the application and understand them visually. (Input: Cooking recipes and cooking instruction videos) (Output: Recipe and video data sent to the user's device) is generated.

[0635] Step 7:

[0636] When the user selects the best recipe from the suggestions, the device sends the order data to an external food delivery service. For example, it orders the food from a nearby restaurant via FoodDeliveryService. (Input: Selected recipe) (Output: Order data to external service) is generated.

[0637] Step 8:

[0638] The server tracks the delivery status of the order in real time via the food delivery service's API. Every time the delivery status information is updated, the server sends it to the terminal and notifies the user. (Input: order data and delivery status information) (Output: real-time delivery status notification) is generated.

[0639] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0640] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0641] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0642] [Second embodiment]

[0643] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0644] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0645] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0646] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0647] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0648] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0649] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0650] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0651] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0652] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0653] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0654] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0655] The present invention is a system that proposes cooking recipes using ingredients in a user's refrigerator and generates cooking videos for those recipes. This system allows users to efficiently create new cooking menus. It also allows users to easily purchase additional ingredients as needed to increase the variety of dishes. The specific program processing of this system is described below.

[0656] System Overview

[0657] 1. Ingredient input method: The user launches the application on their smartphone or PC and inputs the ingredients they have in their refrigerator. This ingredient information is converted to JSON format by the device and sent to the server.

[0658] 2. Recipe generation: The server analyzes the received ingredient information and passes the data to the generation AI to generate a cooking recipe. For example, based on ingredient information such as "chicken, cabbage, and carrots," the generation AI generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[0659] 3. Video generation means: The server provides the generated recipe information to the video generation AI, which generates a video showing the cooking steps in a visually easy-to-understand format. The generated video file is returned to the server.

[0660] 4. User interface: The server sends the acquired recipe information and the generated video to the user's device and displays them through the user interface. The user can view the suggested recipes and videos and select their favorite dishes.

[0661] 5. E-commerce method: If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The user is notified of the delivery status in real time.

[0662] Specific examples

[0663] As a specific example, consider the case where a user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator.

[0664] 1. User: Opens the app on his smartphone and enters the ingredients in his refrigerator: "chicken," "cabbage," and "carrots."

[0665] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[0666] 3. Server: Analyzes the received ingredient list, inputs "chicken, cabbage, carrots" into the generation AI, and generates the following recipes ("stir-fried chicken" and "cabbage and carrot salad").

[0667] 4. Video generation method: Request the video generation AI to generate a cooking video for the above recipe. The generated video file is returned to the server.

[0668] 5. Server: Sends the generated recipes and videos to the user's device.

[0669] 6. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[0670] 7. User: Review suggested recipes and videos and select "Chicken Stir-fry."

[0671] 8. E-commerce method: If the user does not have the necessary additional ingredients (e.g., soy sauce), the user requests to purchase the additional ingredients through the app. The device sends the list of ingredients to the e-commerce site.

[0672] 9. Server: Retrieves soy sauce inventory and price information from the e-commerce site's API and presents it to the user.

[0673] 10. User: Confirms purchase.

[0674] 11. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[0675] 12. E-commerce: Process orders and initiate logistics processes in a short time.

[0676] 13. Server: Notifies the user of delivery status in real time.

[0677] 14. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[0678] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, reduces food waste, and improves the user experience.

[0679] The processing flow will be explained below.

[0680] Step 1:

[0681] User: Launches the application on their smartphone or PC and accesses the screen where they can input the ingredients in their refrigerator.

[0682] Step 2:

[0683] User: Enter the ingredients they currently have on hand, such as "chicken," "cabbage," and "carrots," into the text field.

[0684] Step 3:

[0685] Terminal: Convert the input ingredient list into JSON format and send it to the server.

[0686] Step 4:

[0687] Server: Analyzes the received ingredient list and converts it into a format that can be passed as input data to the generation AI.

[0688] Step 5:

[0689] Server: Pass the ingredients list to the generation AI and have it generate candidate cooking recipe information.

[0690] Step 6:

[0691] Generative AI: Based on a list of ingredients, it generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[0692] Step 7:

[0693] Server: Receives the generated recipe information and provides the video generation AI with cooking instructions and a list of necessary ingredients.

[0694] Step 8:

[0695] Video generation AI: Based on recipe information, it simulates cooking steps and generates visually easy-to-understand cooking videos.

[0696] Step 9:

[0697] Video generation AI: Returns the generated video file to the server.

[0698] Step 10:

[0699] Server: Sends the recipe information and generated video together to the user's device.

[0700] Step 11:

[0701] Terminal: Analyzes the recipe information and video sent from the server and formats them for display on the user interface.

[0702] Step 12:

[0703] Device: The user interface displays recipes such as "Stir-fried chicken" and "Cabbage and carrot salad," along with cooking videos for each recipe that can be played.

[0704] Step 13:

[0705] User: Review suggested recipes and videos and select the dish they prefer.

[0706] Step 14:

[0707] User: Sees an ingredient needed for a selected recipe that is not in the refrigerator (e.g., soy sauce) and asks to purchase it within the app.

[0708] Step 15:

[0709] Terminal: Convert the list of ingredients desired for purchase into JSON format and send it to the EC site's API.

[0710] Step 16:

[0711] Server: Obtains food inventory and price information from the EC site's API and presents it to the user.

[0712] Step 17:

[0713] User: Confirms purchase.

[0714] Step 18:

[0715] Terminal: Generates a request to send purchase confirmation information to the e-commerce site's API via the server.

[0716] Step 19:

[0717] Server: Sends the received request to the e-commerce site and confirms the order.

[0718] Step 20:

[0719] E-commerce sites: process orders and initiate logistics processes quickly.

[0720] Step 21:

[0721] Server: Receives real-time delivery status information from the e-commerce site and notifies the user.

[0722] Step 22:

[0723] User: Check delivery status and wait for additional ingredients to arrive.

[0724] Step 23:

[0725] User: When additional ingredients arrive, check their arrival on the device and proceed with cooking preparations.

[0726] Step 24:

[0727] User: Starts cooking the selected dish using the provided recipe and video.

[0728] The above are the specific processing steps of the system according to the present invention. This process allows users to efficiently use ingredients in their refrigerators and enjoy a variety of dishes.

[0729] Example 1

[0730] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0731] In modern households, it is difficult to efficiently use ingredients in the refrigerator to create new dishes. Furthermore, there is a lack of convenient ways to purchase additional ingredients, which limits the variety of dishes available. Furthermore, there is a lack of ways to clearly communicate cooking steps, which can lead to users making mistakes.

[0732] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0733] In this invention, the server includes a means for inputting ingredient information, a generation AI means for analyzing the input ingredient information and generating candidate recipes, a video generation means for generating cooking procedure videos based on the generated recipes, a user interface means for displaying the generated recipes and cooking procedure videos, an e-commerce means for purchasing additional ingredients, and a means for using prompts when generating candidate recipes. This allows users to efficiently use ingredients in their refrigerators, easily create new dishes, and easily purchase additional ingredients. Furthermore, the cooking procedures can be visually understood through the videos, reducing the risk of making mistakes in cooking procedures.

[0734] The "means for inputting ingredient information" is a means by which a user inputs information about ingredients stored in the refrigerator and transmits that information to the system.

[0735] The "generative AI means" is an artificial intelligence means for analyzing input ingredient information, generating appropriate prompt sentences, and generating cooking recipes based on these.

[0736] The "video generation means" is a means for generating a video including cooking procedures based on the generated cooking recipe.

[0737] The "user interface means" is a means for displaying the generated cooking recipe and cooking procedure video to the user.

[0738] An "electronic commerce means" is a means for purchasing additional ingredients that are needed, and serves to obtain inventory and price information and assist with the purchasing process.

[0739] "Method using prompt sentences" refers to a method of generating a cooking recipe by inputting a prompt sentence created based on ingredient information into a generation AI.

[0740] This invention is a system that proposes cooking recipes using ingredients in a user's refrigerator and generates cooking videos for those recipes. This system allows users to efficiently create new cooking menus. It also allows users to easily purchase additional ingredients as needed to increase the variety of dishes.

[0741] System Overview

[0742] 1. Ingredient input method:

[0743] The user launches the application on their smartphone or PC and manually inputs the ingredients in their refrigerator. This ingredient information is converted to JSON format by the device and sent to the server.

[0744] 2. Generation AI means:

[0745] The server analyzes the received ingredient information and passes the data to the generation AI to generate a cooking recipe. Based on ingredient information such as "chicken, cabbage, and carrots," the generation AI generates multiple recipes using the following example prompt:

[0746] Prompt Sentence Examples

[0747] Please suggest a recipe that can be made using the following ingredients:

[0748] chicken meat

[0749] cabbage

[0750] Carrots

[0751] The generation AI takes the above prompt as input and generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[0752] 3. Video generation method:

[0753] The server then provides the generated recipe information to a video generation AI, which generates a video with cooking instructions in a visually easy-to-understand format. A deep learning-based video generation model, such as PaddleGAN, is used to generate the video. The generated video file is then returned to the server.

[0754] 4. User Interface Means:

[0755] The server sends the acquired recipe information and the generated video to the user's device and displays them through a user interface. The user can view the suggested recipes and videos and select their favorite dishes.

[0756] 5. Electronic commerce instruments:

[0757] If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The delivery status is notified to the user in real time.

[0758] Specific examples

[0759] A specific example will be described in which the user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator.

[0760] 1. User: Opens the smartphone app and manually enters the ingredients in the refrigerator: "chicken," "cabbage," and "carrots."

[0761] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[0762] 3. Server: Analyzes the received ingredient list, inputs "chicken, cabbage, carrots" into the generation AI, and generates the following recipe.

[0763] Stir-fried chicken

[0764] Cabbage and carrot salad

[0765] 4. Server: Requests the video generation AI to generate a cooking video for the above recipe, and returns the generated video file to the server.

[0766] 5. Server: Sends the generated recipes and videos to the user's device.

[0767] 6. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[0768] 7. User: Review suggested recipes and videos and select "Chicken Stir-fry."

[0769] 8. E-commerce method: If the user does not have the necessary additional ingredients (e.g., soy sauce), the user requests to purchase the additional ingredients through the app. The device sends the list of ingredients to the e-commerce site.

[0770] 9. Server: Retrieves inventory and price information from the e-commerce site's API and presents it to the user.

[0771] 10. User: Confirms purchase.

[0772] 11. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[0773] 12. Server: Notifies the user of delivery status in real time.

[0774] 13. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[0775] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, reduces food waste, and improves the user experience.

[0776] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0777] Program processing flow

[0778] Step 1: Enter ingredient information

[0779] User: Launches the application on their smartphone or PC and manually inputs the ingredients they have in their refrigerator. They add ingredients such as "chicken," "cabbage," and "carrots" to the input fields and clicks the submit button.

[0780] Input: Information about ingredients in the refrigerator (e.g., "chicken," "cabbage," "carrots")

[0781] Terminal: Convert the input ingredient information into JSON format.

[0782] Output: JSON formatted ingredient information

[0783] json

[0784] {

[0785] "ingredients": ["chicken", "cabbage", "carrot"]

[0786] }

[0787] What happens: The device collects user-entered information and executes code to convert it into a JSON object, for example, using JavaScript or Python program code.

[0788] Step 2: Submit ingredient information

[0789] Terminal: Sends the converted JSON formatted ingredient information to the server.

[0790] Input: JSON formatted ingredient information

[0791] Output: Ingredient information sent to the server

[0792] What it does: Executes code to create an HTTP POST request and send it to the specified URL on the server.

[0793] Step 3: Recipe generation

[0794] Server: Analyze the received JSON formatted ingredient information.

[0795] Input: JSON formatted ingredient information

[0796] Output: A list of ingredients

[0797] What it does: Uses a JSON parsing library to get the list of ingredients.

[0798] Server: Enters prompt text into the generation AI based on the analyzed data.

[0799] Input: List of ingredients

[0800] Generative AI: Generates cooking recipes based on prompts.

[0801] Prompt Sentence Examples

[0802] Please suggest a recipe that can be made using the following ingredients:

[0803] chicken meat

[0804] cabbage

[0805] Carrots

[0806] Output: Multiple cooking recipes (e.g. "Stir-fried chicken" and "Cabbage and carrot salad")

[0807] Specific operation: Send a prompt to the generation AI (e.g., GPT model) and obtain the generated recipe list.

[0808] Step 4: Save the recipe information

[0809] Server: Stores the generated recipe information in a database.

[0810] Input: Multiple cooking recipes

[0811] Output: Recipe information stored in the database

[0812] What it does: Executes code to add recipe information to the database using an SQL INSERT statement.

[0813] Step 5: Generate cooking videos

[0814] Server: Provides recipe information to the video generation AI.

[0815] Input: Cooking recipe

[0816] Output: Cooking videos generated by video generation AI

[0817] Specific operation: Create an HTTP POST request and send recipe data to the video generation AI API.

[0818] Video generation AI: Generates videos including cooking instructions based on the provided recipe information.

[0819] Input: Cooking recipe

[0820] Output: Cooking videos

[0821] What it does: Runs code that uses a deep learning model to generate a series of image frames and stitches them together into a video.

[0822] Step 6: Submit your recipe and video

[0823] Server: Sends the generated recipe information and cooking videos to the user's device.

[0824] Input: Cooking recipes, cooking videos

[0825] Output: Recipe information and video sent to the user's device

[0826] Specific behavior: Executes code to send data to the user's device via an HTTP GET or POST request.

[0827] Step 7: Display in the user interface

[0828] Terminal: Parses the received data and displays it in the application's user interface.

[0829] Input: Cooking recipes, cooking videos

[0830] Output: Recipe information and video displayed in the user interface

[0831] Specific operation: Analyzes the received JSON data, displays the recipe title, ingredients, and cooking steps in text format, and simultaneously executes code to play the cooking video on a video player.

[0832] Step 8: Purchase additional ingredients

[0833] User: If an additional ingredient (e.g. soy sauce) is needed, the user wishes to purchase the additional ingredient in the app. Click the purchase button and select the required ingredient.

[0834] Terminal: Send the list of ingredients you wish to purchase in JSON format to the e-commerce site.

[0835] Input: List of ingredients you wish to purchase

[0836] Output: Ingredient list sent to the e-commerce site

[0837] What it does: Executes code to create an HTTP POST request and send it to the e-commerce site's API.

[0838] Step 9: Get inventory and pricing information

[0839] Server: Obtains inventory and price information from the e-commerce site's API.

[0840] Input: List of ingredients you wish to purchase

[0841] Output: Stock and price information obtained from the e-commerce site

[0842] Specific behavior: Executes code to retrieve information from the e-commerce site's API via an HTTP GET request and receive it in JSON format.

[0843] Server: Presents stock and price information to the user. Sends data to the device and displays it on the user's screen.

[0844] Input: Stock and price information obtained from the e-commerce site

[0845] Output: Stock and pricing information displayed to the user.

[0846] What it does: Sends data to the device and executes code to display it in the user interface.

[0847] Step 10: Confirm purchase and process order

[0848] User: Confirms purchase by clicking the Confirm button on the screen.

[0849] Input: Purchase confirmation instructions

[0850] Output: Confirmed information

[0851] What it does: Receives input from the user interface and executes code to send purchase confirmation information to the server.

[0852] Server: Receives purchase confirmation information, sends it to the EC site's API, confirms the order, and begins the shipping process.

[0853] Input: Purchase confirmation information

[0854] Output: Order information for the e-commerce site

[0855] What it does: Executes code to create an HTTP POST request and send it to the e-commerce site's API.

[0856] Step 11: Delivery Status Notification

[0857] Server: Notify the user of the delivery status in real time, for example via push notifications or in-app messages.

[0858] Input: Delivery status information

[0859] Output: User notification

[0860] What it does: Executes code to notify the user of delivery status using a push notification server or an in-app notification system.

[0861] As described above, this system efficiently utilizes ingredients in the user's refrigerator and provides new cooking recipes and cooking videos, thereby increasing the variety of dishes available and simplifying the purchase of additional ingredients.

[0862] (Application example 1)

[0863] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0864] Conventional recipe suggestion systems offer limited options for efficiently preparing dishes using ingredients users already have, and lack content that allows users to visually understand cooking methods. This makes it time-consuming and labor-intensive for users to create new recipes, making it difficult to efficiently utilize ingredients. Purchasing additional ingredients is also time-consuming and often burdensome for users. This limits the variety of dishes available and leads to problems such as wasted ingredients.

[0865] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0866] In this invention, the server includes a means for inputting food information, a generative AI model means for analyzing the input food information and generating candidate recipes, a video generation means for generating cooking procedure videos based on the generated recipes, a user interface means for displaying the generated recipes and cooking procedure videos, and an online commerce means for purchasing additional ingredients. This allows users to efficiently use ingredients in their refrigerators and easily create new recipes. Furthermore, by making cooking procedures easier to understand visually through videos, cooking time is reduced and additional ingredients can be purchased more smoothly. This reduces food waste and improves the user experience.

[0867] The "means for inputting food information" is an interface that allows a user to input information about ingredients and other food items in the refrigerator using a terminal that the user owns and transmits this information to the system.

[0868] The "generative AI model means" is a module that uses artificial intelligence technology to generate optimal cooking methods based on input food information.

[0869] The "video generation means" is a technology for generating a visual video of cooking procedures based on the generated recipes, so that users can easily understand how to cook.

[0870] The "user interface means" is an interface that displays the generated recipe and cooking procedure videos to the user and allows the user to view and operate them.

[0871] "Online commerce means" refers to a means by which a user can make an online transaction to purchase additional ingredients that the user needs.

[0872] This system proposes recipes using ingredients that the user has on hand, and generates and distributes videos of the cooking steps, allowing users to efficiently create new recipes and easily purchase any additional ingredients they need.

[0873] System Overview

[0874] The system includes the following means:

[0875] 1. Means for entering food information:

[0876] The user inputs the ingredients in the refrigerator using a device such as a smartphone. This ingredient information is converted into JSON format by the device and sent to the server.

[0877] 2. Generative AI model means:

[0878] The server analyzes the received ingredient information and passes the data to a generative AI to generate recipes. For example, based on ingredient information such as "chicken, cabbage, and carrots," the generative AI model generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[0879] 3. Video generation method:

[0880] The server provides the generated recipe information to the video generation AI, which generates a video with cooking instructions in a visually easy-to-understand format. The generated video file is then returned to the server.

[0881] 4. User Interface Means:

[0882] The server sends the acquired recipe information and the generated video to the user's device and displays them through a user interface. The user can view the suggested recipes and videos and select their favorite dishes.

[0883] 5. Online commerce methods:

[0884] If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The delivery status is notified to the user in real time.

[0885] Program processing

[0886] The system uses the following hardware and software:

[0887] Hardware

[0888] Smartphones and tablets (user devices)

[0889] Cloud Server

[0890] software

[0891] On the server side, recipe generation AI (e.g., GPT-4) and video generation AI (e.g., DALL-E and Deep Video) are used.

[0892] On the client side, a communication library (e.g., requests) is used to send and receive data in JSON format.

[0893] A standard web front-end framework (e.g., React, Angular, etc.) is used to display the user interface.

[0894] Specific examples

[0895] As a specific example, consider the case where the user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator.

[0896] Prompt Sentence Examples

[0897] Suggest a simple and delicious recipe using the following ingredients you have in your fridge:

[0898] chicken meat

[0899] cabbage

[0900] Carrots

[0901] The server passes this prompt to the generation AI, which generates a recipe like this:

[0902] "Stir-fried chicken" and "Cabbage and carrot salad"

[0903] The server then passes this recipe information to a video generation AI, which generates a video of the cooking steps. The generated data is stored in cloud data storage and sent to the user's device. Users can view these recipes and videos through a smartphone app and begin cooking based on the recipe they select. If additional ingredients are required, they can easily purchase them through the e-commerce site.

[0904] This system allows users to make the most of the ingredients they have, reduce waste, and easily enjoy new cooking menus.

[0905] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0906] Step 1:

[0907] The user launches the app on their smartphone and inputs the ingredients they have in their refrigerator. The user enters the specific ingredients they have (e.g., chicken, cabbage, carrots) into the app. This information is converted into a data format (e.g., JSON) and sent to the server.

[0908] Input: Ingredient information entered by the user

[0909] Output: Ingredient information in JSON format

[0910] Step 2:

[0911] The server analyzes the received ingredient information and passes the data to the generative AI model. The server then generates a prompt based on the ingredient information and inputs it into the generative AI model (e.g., GPT-4). During this analysis, the input ingredient information is formatted appropriately.

[0912] Input: JSON formatted ingredient information

[0913] Output: A prompt to be passed to the generative AI model

[0914] Step 3:

[0915] The server uses a generative AI model to generate multiple recipes based on the input prompt, which analyzes and generates several candidate recipes based on the user's ingredients.

[0916] Input: prompt statement

[0917] Output: Multiple generated recipes

[0918] Step 4:

[0919] The server provides the generated recipe information to the video generation AI, which then generates cooking instructions as videos. The video generation AI then generates video files that visually explain the steps of each recipe in an easy-to-understand format.

[0920] Input: Generated recipe information

[0921] Output: Video file visualizing the cooking steps

[0922] Step 5:

[0923] The server sends the generated recipes and videos to the user's device, stores them in cloud storage, and delivers them to the user's app in real time.

[0924] Input: Generated recipes and videos

[0925] Output: User device where recipes and videos are delivered

[0926] Step 6:

[0927] Users can browse suggested recipes and videos through the app and select their preferred dish. Users can then select one of the multiple options on the app and see how to cook it.

[0928] Input: Distributed recipes and videos

[0929] Output: User selected recipe

[0930] Step 7:

[0931] If the user wishes to purchase additional ingredients, the device sends a list of the ingredients they wish to purchase to the EC site. The device then creates a list of the ingredients needed based on the selected recipe and sends the data to the EC site.

[0932] Input: A list of additional ingredients based on the recipe selected by the user

[0933] Output: Ingredient list sent to the e-commerce site

[0934] Step 8:

[0935] The server calls the API of the e-commerce site, obtains stock and price information for each ingredient, and presents it to the user. The server analyzes the API response and presents the information in a format that is easy for the user to understand.

[0936] Input: List of additional ingredients sent

[0937] Output: Retrieved inventory and price information

[0938] Step 9:

[0939] The user confirms the purchase procedure through the app and confirms the purchase. The app receives the user's input and sends it to the server.

[0940] Input: User's purchase confirmation information

[0941] Output: Purchase confirmation information

[0942] Step 10:

[0943] The server sends the purchase confirmation information to the e-commerce site's API to complete the order process. The server also sends the order information via the API to start the logistics process.

[0944] Input: User's purchase confirmation information

[0945] Output: Order information sent to the e-commerce site

[0946] Step 11:

[0947] The server notifies the user of the delivery status in real time. The server obtains the delivery status from the EC site and notifies the user.

[0948] Input: Delivery status from the e-commerce site

[0949] Output: Real-time notification of delivery status

[0950] This process allows users to effectively use the ingredients they have and smoothly purchase any additional ingredients they need. The generated cooking video also reduces the time and effort required for cooking.

[0951] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0952] The present invention is a system that suggests cooking recipes using ingredients in the user's refrigerator and generates cooking videos for those recipes. By further combining this system with an emotion engine that recognizes the user's emotions, it is possible to suggest recipes and videos that correspond to the user's emotional state. This system not only enables users to efficiently create new cooking menus, but also provides a more satisfying experience by receiving suggestions that match their emotions. Specific program processing for this system is described below.

[0953] System Overview

[0954] 1. Ingredient input method: The user launches the application on their smartphone or PC and inputs the ingredients they have in their refrigerator. This ingredient information is converted to JSON format by the device and sent to the server.

[0955] 2. Recipe generation: The server analyzes the received ingredient information and passes the data to the generation AI to generate a cooking recipe. For example, based on ingredient information such as "chicken, cabbage, and carrots," it generates recipes such as "stir-fried chicken" or "cabbage and carrot salad."

[0956] 3. Emotion Engine: When a user is using the application, sensors such as a camera and microphone are used to analyze the user's facial expressions and voice to recognize their emotional state. This emotional information is then sent to the server.

[0957] 4. Emotion-based suggestions: The server analyzes the emotion information received from the emotion engine and adjusts the recipe suggestions based on the user's current emotional state. For example, if the user is feeling stressed, it will suggest dishes with a relaxing effect.

[0958] 5. Video generation method: The server provides recipe information and emotion information to the video generation AI, which generates a video with cooking instructions that match the emotion and is easy to understand visually. The generated video file is returned to the server.

[0959] 6. User interface means: The server sends the acquired recipe information, emotion information, and generated video to the user's device and displays them through the user interface. The user can view the suggested recipes and videos and choose their favorite dishes.

[0960] 7. E-commerce method: If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The delivery status is notified to the user in real time.

[0961] Specific examples

[0962] As a specific example, consider a case where a user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator, and the emotion engine recognizes that the user is feeling stressed.

[0963] 1. User: Opens the app on his smartphone and enters the ingredients in his refrigerator: "chicken," "cabbage," and "carrots."

[0964] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[0965] 3. Server: Analyzes the received ingredient list, inputs "chicken, cabbage, carrots" into the generation AI, and generates the following recipes ("stir-fried chicken" and "cabbage and carrot salad").

[0966] 4. Emotion engine: Analyzes the user's facial expressions and voice and notifies the server that the user is feeling stressed.

[0967] 5. Server: Analyzes information from the emotion engine and prioritizes recipes that have a relaxing effect.

[0968] 6. Video generation method: The video generation AI is requested to generate a cooking video for the "cabbage and carrot salad" recipe, which is particularly relaxing from the above recipes. The generated video file is returned to the server.

[0969] 7. Server: Sends the generated recipes and videos to the user's device.

[0970] 8. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[0971] 9. User: Review suggested relaxing recipes and videos and choose to cook.

[0972] 10. E-commerce method: If the user does not have the necessary additional ingredients (e.g., olive oil), the user requests to purchase the additional ingredients through the app. The device sends the list of ingredients to the e-commerce site.

[0973] 11. Server: Retrieves olive oil inventory and price information from the e-commerce site's API and presents it to the user.

[0974] 12. User: Confirms purchase.

[0975] 13. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[0976] 14. E-commerce: Process orders and initiate logistics processes in a short time.

[0977] 15. Server: Notifies the user of delivery status in real time.

[0978] 16. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[0979] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, and provides a more satisfying cooking experience by making suggestions based on the user's emotional state.

[0980] The processing flow will be explained below.

[0981] Step 1:

[0982] User: Launches the application on their smartphone or PC and accesses the screen where they can input the ingredients in their refrigerator.

[0983] Step 2:

[0984] User: Enter the ingredients they currently have on hand, such as "chicken," "cabbage," and "carrots," into the text field.

[0985] Step 3:

[0986] Terminal: Convert the input ingredient list into JSON format and send it to the server.

[0987] Step 4:

[0988] Server: Analyzes the received ingredient list and converts it into a format that can be passed as input data to the generation AI.

[0989] Step 5:

[0990] Server: Pass the ingredients list to the generation AI and have it generate candidate cooking recipe information.

[0991] Step 6:

[0992] Generative AI: Based on a list of ingredients, it generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[0993] Step 7:

[0994] Server: Receives the generated recipe information.

[0995] Step 8:

[0996] Emotion engine: When a user is using an application, the system captures the user's facial expressions and voice data via the camera and microphone, and analyzes this to recognize the user's emotional state.

[0997] Step 9:

[0998] Emotion engine: Sends the recognized emotional state to the server. For example, if the user is feeling stressed, it sends the data of the stress state.

[0999] Step 10:

[1000] Server: Analyzes the emotional information received from the emotion engine and adjusts recipe suggestions based on the user's current emotional state, for example, prioritizing dishes with a relaxing effect.

[1001] Step 11:

[1002] Server: Provides recipe information and emotional information to the video generation AI and asks it to generate a cooking instruction video that matches the emotion.

[1003] Step 12:

[1004] Video generation AI: Based on recipe information, it simulates cooking steps and generates visually easy-to-understand cooking videos.

[1005] Step 13:

[1006] Video generation AI: Returns the generated video file to the server.

[1007] Step 14:

[1008] Server: Sends the recipe information and generated video together to the user's device.

[1009] Step 15:

[1010] Terminal: Analyzes the recipe information and video sent from the server and formats them for display on the user interface.

[1011] Step 16:

[1012] Device: The user interface displays recipes such as "Stir-fried chicken" and "Cabbage and carrot salad," along with cooking videos for each recipe that can be played.

[1013] Step 17:

[1014] User: Review suggested recipes and videos and select the dish they prefer.

[1015] Step 18:

[1016] User: Sees an ingredient needed for a selected recipe that is not in the refrigerator (e.g., soy sauce) and asks to purchase it within the app.

[1017] Step 19:

[1018] Terminal: Convert the list of ingredients desired for purchase into JSON format and send it to the EC site's API.

[1019] Step 20:

[1020] Server: Obtains food inventory and price information from the EC site's API and presents it to the user.

[1021] Step 21:

[1022] User: Confirms purchase.

[1023] Step 22:

[1024] Terminal: Generates a request to send purchase confirmation information to the e-commerce site's API via the server.

[1025] Step 23:

[1026] Server: Sends the received request to the e-commerce site and confirms the order.

[1027] Step 24:

[1028] E-commerce sites: process orders and initiate logistics processes quickly.

[1029] Step 25:

[1030] Server: Receives real-time delivery status information from the e-commerce site and notifies the user.

[1031] Step 26:

[1032] User: Check delivery status and wait for additional ingredients to arrive.

[1033] Step 27:

[1034] User: When additional ingredients arrive, check their arrival on the device and proceed with cooking preparations.

[1035] Step 28:

[1036] User: Starts cooking the selected dish using the provided recipe and video.

[1037] The above are the specific processing steps of the system according to the present invention. This process not only allows users to efficiently use the ingredients in their refrigerator to enjoy a variety of dishes, but also provides a more satisfying cooking experience by receiving recipe suggestions that match their emotions.

[1038] Example 2

[1039] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1040] Conventional recipe suggestion systems are unable to consider the user's emotional state when proposing recipes, making it difficult to enhance the user's psychological satisfaction. Furthermore, they lack a means to provide easy-to-understand visual instructions for cooking, making them difficult to use, especially for beginners. Another issue is the complicated process of purchasing additional ingredients, which makes it difficult to start cooking smoothly.

[1041] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1042] In this invention, the server includes a means for inputting ingredient information, an artificial intelligence means for analyzing the input ingredient information and generating candidate recipes, an emotion engine means for recognizing the user's emotional state and adjusting the cooking recipe based on the recognized emotional information, a means for generating cooking procedure videos based on the generated cooking recipe, a user interface means for displaying the generated cooking recipe and cooking procedure videos, and an electronic commerce means for purchasing any additional ingredients required. This allows the user to receive recipe suggestions suited to their emotional state and to proceed with cooking while referring to the visually easy-to-understand cooking videos. It also makes it easy to purchase any additional ingredients required.

[1043] The "means for inputting ingredient information" is an interface that allows users to input ingredients in the refrigerator as text or images via a smartphone or PC application.

[1044] The "artificial intelligence means" is a system that analyzes input ingredient information and generates candidate cooking recipes using a generative AI model.

[1045] The "emotion engine means" is a function that analyzes the user's facial expressions and voice, recognizes the user's emotional state, and adjusts the cooking recipe based on the emotional information.

[1046] The "means for generating cooking procedure videos" is a technology for generating cooking procedure videos that are visually easy to understand based on the generated cooking recipe.

[1047] The "user interface means" is an interface for displaying the generated cooking recipe and cooking procedure video to the user.

[1048] "Electronic commerce means" refers to a system that provides online shopping functionality for purchasing additional ingredients needed.

[1049] The present invention is a system that suggests cooking recipes using ingredients in the user's refrigerator and generates cooking videos for those recipes. Furthermore, by combining it with an emotion engine that recognizes the user's emotional state, it suggests recipes and videos according to that emotional state. This system allows users to efficiently create new cooking menus, and by receiving suggestions that match their emotions, it increases their satisfaction. The system's program processing is explained in detail below.

[1050] System Overview

[1051] Entering ingredient information

[1052] Users enter information about ingredients in their refrigerators through a smartphone or PC application. Input methods include text entry and uploading images taken with a camera. The entered information is converted to JSON format by the device and sent to the server.

[1053] Recipe Generation

[1054] The server analyzes the received JSON-formatted ingredient information and sends a prompt to the generative AI model (e.g., GPT-3) saying, "Please suggest a cooking recipe that includes chicken, cabbage, and carrots." The generative AI model generates multiple recipes based on the received prompt.

[1055] emotion recognition

[1056] When a user is using an application, sensors such as a camera and microphone are used to record facial expressions and voice. The device then uses the recorded data to analyze the user's emotional state using an emotion recognition engine (e.g., Microsoft Azure Emotion API). The analyzed emotional information is converted into JSON format and sent to the server.

[1057] Emotion-based recipe suggestions

[1058] The server analyzes the received emotional information and adjusts the recipe suggestions based on the user's current emotional state. For example, if the user is feeling stressed, it will prioritize suggestions of cooking recipes that have a relaxing effect. As a specific example, it generates "stir-fried chicken" and "cabbage and carrot salad," and prioritizes displaying the "cabbage and carrot salad," which has the effect of relieving stress.

[1059] Cooking video generation

[1060] The server requests a video generation AI (e.g., Synthesia) to generate a video explaining the cooking steps for the specified recipe based on the emotional state and recipe information. The generated video file is sent back to the server and provided to the user.

[1061] User display and operation

[1062] The server sends the acquired recipe information, emotion information, and generated cooking videos to the user's device, which analyzes the information and displays it through a user interface. The user can then check and operate the suggested recipes and videos.

[1063] Purchase additional ingredients

[1064] If the user does not have the necessary ingredients to prepare a suggested recipe, they can request to purchase the ingredients through the app. The device sends a list of the ingredients they wish to purchase to the EC site via an e-commerce transaction method. The server retrieves inventory and price information from the EC site's API and presents it to the user. Once the user confirms the purchase, the server sends the order information to the EC site's API and confirms the order. The user is notified of the delivery status in real time.

[1065] Specific examples

[1066] As a specific example, consider a case where a user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator, and the emotion engine recognizes that the user is feeling stressed.

[1067] 1. User: Opens the app on his smartphone and enters the ingredients in his refrigerator: "chicken," "cabbage," and "carrots."

[1068] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[1069] 3. Server: Analyzes the received list of ingredients, inputs "chicken, cabbage, carrots" into a generation AI (e.g., GPT-3), and generates recipes such as the following ("stir-fried chicken" and "cabbage and carrot salad").

[1070] 4. Emotion engine: Analyzes the user's facial expressions and voice and notifies the server that the user is feeling stressed.

[1071] 5. Server: Analyzes information from the emotion engine and prioritizes recipes that have a relaxing effect.

[1072] 6. Video generation method: A video generation AI (e.g., Synthesia) is requested to generate a cooking video for the "cabbage and carrot salad" recipe, which is particularly relaxing from the above recipes. The generated video file is returned to the server.

[1073] 7. Server: Sends the generated recipes and videos to the user's device.

[1074] 8. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[1075] 9. User: Review suggested relaxing recipes and videos and choose to cook.

[1076] 10. E-commerce method: If you don't have the additional ingredients you need (e.g. olive oil), submit a list of ingredients you want to purchase to the e-commerce site.

[1077] 11. Server: Retrieves olive oil inventory and price information from the e-commerce site's API and presents it to the user.

[1078] 12. User: Confirms purchase.

[1079] 13. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[1080] 14. E-commerce website: Processes orders and initiates the delivery process.

[1081] 15. Server: Notifies the user of delivery status in real time.

[1082] 16. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[1083] Prompt Sentence Examples

[1084] An example of a prompt sentence to input into the generative AI model is, "Please suggest a relaxing cooking recipe that includes chicken, cabbage, and carrots."

[1085] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, and provides a more satisfying cooking experience by making suggestions based on the user's emotional state.

[1086] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1087] Step 1:

[1088] The user enters ingredient information

[1089] Users launch the application on their smartphone or PC and enter information about the ingredients in their refrigerator. Specifically, they enter "chicken," "cabbage," and "carrots" in the text input field and press the send button. Alternatively, they can take a photo of the ingredients and upload it.

[1090] input:

[1091] Ingredient information (text or image)

[1092] output:

[1093] Ingredient information is saved on the device

[1094] Step 2:

[1095] The device analyzes the ingredient information and sends it to the server.

[1096] The device extracts the names of ingredients from the input text data as is, or from image data using an image recognition engine (e.g., Google Cloud Vision API). The extracted names are converted to JSON format and sent to the server.

[1097] input:

[1098] Ingredient information (text or image)

[1099] Data processing / calculation:

[1100] Convert text information (directly into JSON)

[1101] Recognition processing using an image analysis engine (image → text information)

[1102] output:

[1103] JSON formatted ingredient information

[1104] Step 3:

[1105] The server analyzes the sent ingredient information and sends a prompt to the generative AI model.

[1106] The server analyzes the received JSON-formatted ingredient information and sends a prompt to the generative AI model (e.g., GPT-3) such as, "Please suggest a cooking recipe that includes chicken, cabbage, and carrots."

[1107] input:

[1108] JSON formatted ingredient information

[1109] Data processing / calculation:

[1110] Prompt statement generation and sending (creating a prompt statement and sending it)

[1111] output:

[1112] Prompt to send to the generative AI model

[1113] Step 4:

[1114] A generative AI model generates cooking recipes based on prompts

[1115] The generative AI model receives a prompt and generates potential recipes based on the prompt, such as "stir-fried chicken" or "cabbage and carrot salad."

[1116] input:

[1117] Prompt statement

[1118] Data Calculation:

[1119] Recipe generation using natural language processing

[1120] output:

[1121] Cooking recipe information

[1122] Step 5:

[1123] The server receives the generated recipe and obtains the emotion information.

[1124] The server stores the generated recipe information and then prepares to acquire the user's emotional information, specifically, issues an instruction to the terminal to acquire the user's current emotional state.

[1125] input:

[1126] Cooking recipe information

[1127] output:

[1128] Instructions for obtaining emotional information

[1129] Step 6:

[1130] The device records the user's facial expressions and voice and analyzes them using an emotion engine.

[1131] When a user is using an application, the device's camera and microphone are used to record facial expressions and voice. The device then uses the recorded data to analyze the user's emotional state using an emotion recognition engine (e.g., Microsoft Azure Emotion API).

[1132] input:

[1133] Facial expression data, voice data

[1134] Data processing / calculation:

[1135] Analysis by emotion recognition engine

[1136] output:

[1137] Emotion information in JSON format

[1138] Step 7:

[1139] The device sends the analyzed emotional information to the server.

[1140] The device converts the analysis results into emotional information in JSON format and sends it to the server. For example, information such as "Emotion: Stress" is sent.

[1141] input:

[1142] Emotion information (JSON format)

[1143] output:

[1144] Send emotion information to the server

[1145] Step 8:

[1146] The server analyzes the emotional information and adjusts the recipe suggestions.

[1147] The server analyzes the received emotional information and adjusts recipe suggestions based on the user's current emotional state. For example, if a user is feeling stressed, it will prioritize recipes that have a relaxing effect.

[1148] input:

[1149] Emotional information, cooking recipe information

[1150] Data processing / calculation:

[1151] Recipe adjustment based on emotion information

[1152] output:

[1153] Adjusted cooking recipe information

[1154] Step 9:

[1155] The server requests the video generation AI to generate a cooking procedure video.

[1156] Based on the emotional state and recipe information, the server requests a video generation AI (e.g., Synthesia) to generate a video explaining the cooking steps for the specified recipe.

[1157] input:

[1158] Adjusted cooking recipe information

[1159] Data Calculation:

[1160] Video generation by video generation AI

[1161] output:

[1162] Cooking procedure video

[1163] Step 10:

[1164] The server sends the generated recipe and video to the device.

[1165] The server sends the generated recipe information and cooking procedure video to the user's terminal.

[1166] input:

[1167] Cooking procedure videos, adjusted cooking recipe information

[1168] output:

[1169] Send recipe information and videos to your device

[1170] Step 11:

[1171] The device analyzes the received information and displays it on the user interface.

[1172] The device analyzes the recipe information and cooking video sent from the server and displays them on the user interface. The user can then check the suggested recipes and videos and choose their preferred dish.

[1173] input:

[1174] Recipe information, cooking procedure videos

[1175] output:

[1176] Display in the user interface

[1177] Step 12:

[1178] The user performs an operation to purchase any additional ingredients needed.

[1179] If the user does not have the necessary ingredients to prepare the suggested recipe, they can request to purchase the ingredients through the app. They can create a list of ingredients they wish to purchase and send it to their device.

[1180] input:

[1181] Missing ingredient information

[1182] output:

[1183] List of ingredients you want to buy

[1184] Step 13:

[1185] The device sends the list of ingredients desired to be purchased to the e-commerce site and displays the information.

[1186] The terminal sends the list of ingredients desired to be purchased to the EC site via the electronic commerce means. The server retrieves inventory and price information from the EC site's API and presents it to the user.

[1187] input:

[1188] List of ingredients you want to buy

[1189] output:

[1190] Send to EC site and display stock and price information

[1191] Step 14:

[1192] The user confirms the purchase and the server sends the order information to the e-commerce site.

[1193] The user checks the inventory and price information provided and confirms the purchase. The server sends the purchase confirmation information to the EC site's API and confirms the order. The EC site processes the order and starts the delivery process.

[1194] input:

[1195] Purchase confirmation information

[1196] output:

[1197] Send order information to the EC site

[1198] Step 15:

[1199] The server notifies the user of the delivery status in real time.

[1200] The server notifies the user of the delivery status in real time. When the additional ingredients arrive, the user can check the arrival and start cooking. The user can cook according to the recipe while watching the provided video.

[1201] input:

[1202] Delivery Status Information

[1203] output:

[1204] Notify users of delivery status

[1205] (Application example 2)

[1206] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1207] While conventional cooking recommendation systems can suggest recipes based on the ingredients in the refrigerator, they are unable to suggest optimal dishes based on the user's emotional state. This makes it difficult to accurately suggest dishes desired when the user is in a specific emotional state, and there are a lack of means to improve the user experience. Furthermore, there is a lack of functionality to immediately order the suggested dishes and track their delivery status.

[1208] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting ingredient information, artificial intelligence means for analyzing the input ingredient information and generating candidate recipes, means for generating cooking procedure videos based on the generated recipes, user interface means for displaying the generated recipes and cooking procedure videos, e-commerce means for purchasing additional ingredients, an emotion engine for recognizing the user's emotions, means for suggesting recipes based on the user's emotional state, means for ordering the suggested dishes from an external service, and means for tracking the delivery status of the ordered dishes. This enables the server to suggest optimal dishes based on the user's emotional state, and to quickly order the suggested dishes and track their delivery status.

[1209] The "means for inputting ingredient information" refers to the interface or terminal function that allows the user to input ingredient information about ingredients stored in the refrigerator.

[1210] The "artificial intelligence means for analyzing input ingredient information and generating candidate cooking recipes" refers to an artificial intelligence algorithm and calculation mechanism for generating appropriate cooking recipes based on the ingredient information input by the user.

[1211] "Means for generating videos of cooking steps based on generated cooking recipes" refers to technology and devices for creating videos of cooking steps that are visually easy to understand based on cooking recipes generated by AI.

[1212] The "user interface means for displaying the generated cooking recipe and cooking procedure video" refers to an interface and device that allows the user to visually check the generated cooking recipe and cooking procedure video.

[1213] "Electronic commerce means for purchasing additional ingredients" refers to a system and procedure for purchasing ingredients that are in short supply via the Internet.

[1214] An "emotion engine that recognizes user emotions" is a combination of software and hardware that uses a camera and microphone to analyze emotions from the user's facial expressions and voice and recognize those emotions.

[1215] The "means for suggesting recipes according to the user's emotional state" refers to an algorithm and means for selecting and suggesting optimal cooking recipes based on the user's recognized emotions.

[1216] "Means for ordering suggested dishes from an external service" refers to a system and function for ordering the dishes selected by the user from an external food delivery service or the like.

[1217] The "means for tracking the delivery status of an ordered meal" refers to a system and technology for tracking the current delivery status of an ordered meal in real time and notifying the user.

[1218] The present invention is a system that suggests optimal dishes based on a user's emotional state and orders them through an external food delivery service. This system allows users to easily order dishes that match their emotions and track delivery status in real time. Specific embodiments of this system are described below.

[1219] System Overview

[1220] The system of the present invention comprises the following components:

[1221] 1. Ingredient input method:

[1222] Users can input information about ingredients in their refrigerators using an application on their smartphone or PC. Similar applications include smart fridge management apps.

[1223] 2. Artificial Intelligence Means:

[1224] The system analyzes the input ingredient information and generates candidate recipes. This analysis is performed using a generative AI model. For example, if "chicken, cabbage, carrots" is input, recipes such as "stir-fried chicken" and "cabbage and carrot salad" will be generated.

[1225] 3. Video generation method:

[1226] Based on the generated recipe, a video showing the cooking steps is generated. This video generation is also done using an AI model. An intuitive and easy-to-understand video is created for each recipe.

[1227] 4. User Interface Means:

[1228] The generated recipes and cooking video are displayed on the user's device interface, allowing the user to select which dish to cook based on this information.

[1229] 5. Electronic commerce instruments:

[1230] It provides a means for users to purchase additional ingredients they need. When users input the ingredients they are missing, it connects with the API of the e-commerce site to carry out the online purchase process based on that information.

[1231] 6. Emotion Engine:

[1232] The system analyzes the user's facial expressions and voice to recognize their emotional state. This analysis is performed using a camera and microphone, and emotion analysis software such as EmotionEngine is used.

[1233] 7. Emotion-based recipe suggestion method:

[1234] Based on the emotional information obtained from the emotion engine, the system suggests recipes that are optimal for the user's current emotional state. For example, if the user is feeling stressed, the system suggests dishes that have a relaxing effect.

[1235] 8. How to order from third-party services:

[1236] Order the suggested dishes from an external food delivery service, specifically by placing an order with the restaurant via an API such as FoodDeliveryService.

[1237] 9. Delivery status tracking:

[1238] Track your order delivery in real time and get notified every time the delivery status changes.

[1239] Program processing overview

[1240] The server includes a means for inputting ingredient information, an artificial intelligence means for analyzing the input ingredient information and generating candidate recipes, and a means for generating videos of cooking procedures based on the generated recipes, thereby enabling the server to suggest optimal dishes according to the user's emotional state, and to quickly order the dishes and track their delivery status.

[1241] Specific examples

[1242] For example, if a user inputs that they have "chicken," "cabbage," and "carrots" in their refrigerator, and the app analyzes their facial expressions and voice and determines that they are feeling stressed, it will suggest a relaxing "cabbage and carrot salad." When the user selects that recipe, it places an order through an external food delivery service and tracks the progress of the ingredients in real time until they are delivered.

[1243] Prompt Sentence Examples

[1244] "A system that suggests the best meal for a stressed user who has chicken, cabbage, and carrots in the refrigerator, orders the meal, and tracks its delivery."

[1245] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1246] Step 1:

[1247] The user launches the application on their smartphone or PC and enters information about ingredients in the refrigerator. The ingredient information entered by the user is converted to JSON format data by the device and sent to the server. This generates (input: ingredient information entered by the user) (output: ingredient data in JSON format).

[1248] Step 2:

[1249] The server analyzes the received JSON-formatted ingredient information and generates a cooking recipe based on the ingredient information. This process uses a generative AI model. For example, if the data "chicken, cabbage, carrots" is input, the server generates candidate recipes such as "stir-fried chicken" or "cabbage and carrot salad." (Input: JSON-formatted ingredient data) (Output: candidate cooking recipe) is generated.

[1250] Step 3:

[1251] When a user is operating an application, the device's camera and microphone are used to detect the user's facial expressions and voice, which are then analyzed by the emotion engine. The analysis results (e.g., information on the user's stress level) are sent to the server, where (input: user's facial expression data and voice data) (output: emotion analysis results) are generated.

[1252] Step 4:

[1253] The server uses the emotion analysis results received from the emotion engine to prioritize recipe suggestions that match the user's emotional state. In this case, recipes that are expected to have a relaxing effect or improve mood are selected. (Input: Candidate recipes and emotion analysis results) (Output: Optimal recipe suggestions based on emotions) are generated.

[1254] Step 5:

[1255] The server generates a cooking procedure video based on the generated recipe using a video generation means. The generated cooking procedure video is returned to the server. (Input: optimal cooking recipe) (Output: cooking procedure video) is generated.

[1256] Step 6:

[1257] The server sends the generated recipes and cooking instruction videos to the user's device. The user can check these suggestions in the application and understand them visually. (Input: Cooking recipes and cooking instruction videos) (Output: Recipe and video data sent to the user's device) is generated.

[1258] Step 7:

[1259] When the user selects the best recipe from the suggestions, the device sends the order data to an external food delivery service. For example, it orders the food from a nearby restaurant via FoodDeliveryService. (Input: Selected recipe) (Output: Order data to external service) is generated.

[1260] Step 8:

[1261] The server tracks the delivery status of the order in real time via the food delivery service's API. Every time the delivery status information is updated, the server sends it to the terminal and notifies the user. (Input: order data and delivery status information) (Output: real-time delivery status notification) is generated.

[1262] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1263] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1264] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1265] [Third embodiment]

[1266] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1267] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1268] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1269] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1270] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1271] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1272] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1273] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1274] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1275] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1276] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1277] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1278] The present invention is a system that proposes cooking recipes using ingredients in a user's refrigerator and generates cooking videos for those recipes. This system allows users to efficiently create new cooking menus. It also allows users to easily purchase additional ingredients as needed to increase the variety of dishes. The specific program processing of this system is described below.

[1279] System Overview

[1280] 1. Ingredient input method: The user launches the application on their smartphone or PC and inputs the ingredients they have in their refrigerator. This ingredient information is converted to JSON format by the device and sent to the server.

[1281] 2. Recipe generation: The server analyzes the received ingredient information and passes the data to the generation AI to generate a cooking recipe. For example, based on ingredient information such as "chicken, cabbage, and carrots," the generation AI generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[1282] 3. Video generation means: The server provides the generated recipe information to the video generation AI, which generates a video showing the cooking steps in a visually easy-to-understand format. The generated video file is returned to the server.

[1283] 4. User interface: The server sends the acquired recipe information and the generated video to the user's device and displays them through the user interface. The user can view the suggested recipes and videos and select their favorite dishes.

[1284] 5. E-commerce method: If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The user is notified of the delivery status in real time.

[1285] Specific examples

[1286] As a specific example, consider the case where a user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator.

[1287] 1. User: Opens the app on his smartphone and enters the ingredients in his refrigerator: "chicken," "cabbage," and "carrots."

[1288] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[1289] 3. Server: Analyzes the received ingredient list, inputs "chicken, cabbage, carrots" into the generation AI, and generates the following recipes ("stir-fried chicken" and "cabbage and carrot salad").

[1290] 4. Video generation method: Request the video generation AI to generate a cooking video for the above recipe. The generated video file is returned to the server.

[1291] 5. Server: Sends the generated recipes and videos to the user's device.

[1292] 6. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[1293] 7. User: Review suggested recipes and videos and select "Chicken Stir-fry."

[1294] 8. E-commerce method: If the user does not have the necessary additional ingredients (e.g., soy sauce), the user requests to purchase the additional ingredients through the app. The device sends the list of ingredients to the e-commerce site.

[1295] 9. Server: Retrieves soy sauce inventory and price information from the e-commerce site's API and presents it to the user.

[1296] 10. User: Confirms purchase.

[1297] 11. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[1298] 12. E-commerce: Process orders and initiate logistics processes in a short time.

[1299] 13. Server: Notifies the user of delivery status in real time.

[1300] 14. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[1301] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, reduces food waste, and improves the user experience.

[1302] The processing flow will be explained below.

[1303] Step 1:

[1304] User: Launches the application on their smartphone or PC and accesses the screen where they can input the ingredients in their refrigerator.

[1305] Step 2:

[1306] User: Enter the ingredients they currently have on hand, such as "chicken," "cabbage," and "carrots," into the text field.

[1307] Step 3:

[1308] Terminal: Convert the input ingredient list into JSON format and send it to the server.

[1309] Step 4:

[1310] Server: Analyzes the received ingredient list and converts it into a format that can be passed as input data to the generation AI.

[1311] Step 5:

[1312] Server: Pass the ingredients list to the generation AI and have it generate candidate cooking recipe information.

[1313] Step 6:

[1314] Generative AI: Based on a list of ingredients, it generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[1315] Step 7:

[1316] Server: Receives the generated recipe information and provides the video generation AI with cooking instructions and a list of necessary ingredients.

[1317] Step 8:

[1318] Video generation AI: Based on recipe information, it simulates cooking steps and generates visually easy-to-understand cooking videos.

[1319] Step 9:

[1320] Video generation AI: Returns the generated video file to the server.

[1321] Step 10:

[1322] Server: Sends the recipe information and generated video together to the user's device.

[1323] Step 11:

[1324] Terminal: Analyzes the recipe information and video sent from the server and formats them for display on the user interface.

[1325] Step 12:

[1326] Device: The user interface displays recipes such as "Stir-fried chicken" and "Cabbage and carrot salad," along with cooking videos for each recipe that can be played.

[1327] Step 13:

[1328] User: Review suggested recipes and videos and select the dish they prefer.

[1329] Step 14:

[1330] User: Sees an ingredient needed for a selected recipe that is not in the refrigerator (e.g., soy sauce) and asks to purchase it within the app.

[1331] Step 15:

[1332] Terminal: Convert the list of ingredients desired for purchase into JSON format and send it to the EC site's API.

[1333] Step 16:

[1334] Server: Obtains food inventory and price information from the EC site's API and presents it to the user.

[1335] Step 17:

[1336] User: Confirms purchase.

[1337] Step 18:

[1338] Terminal: Generates a request to send purchase confirmation information to the e-commerce site's API via the server.

[1339] Step 19:

[1340] Server: Sends the received request to the e-commerce site and confirms the order.

[1341] Step 20:

[1342] E-commerce sites: process orders and initiate logistics processes quickly.

[1343] Step 21:

[1344] Server: Receives real-time delivery status information from the e-commerce site and notifies the user.

[1345] Step 22:

[1346] User: Check delivery status and wait for additional ingredients to arrive.

[1347] Step 23:

[1348] User: When additional ingredients arrive, check their arrival on the device and proceed with cooking preparations.

[1349] Step 24:

[1350] User: Starts cooking the selected dish using the provided recipe and video.

[1351] The above are the specific processing steps of the system according to the present invention. This process allows users to efficiently use ingredients in their refrigerators and enjoy a variety of dishes.

[1352] Example 1

[1353] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1354] In modern households, it is difficult to efficiently use ingredients in the refrigerator to create new dishes. Furthermore, there is a lack of convenient ways to purchase additional ingredients, which limits the variety of dishes available. Furthermore, there is a lack of ways to clearly communicate cooking steps, which can lead to users making mistakes.

[1355] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1356] In this invention, the server includes a means for inputting ingredient information, a generation AI means for analyzing the input ingredient information and generating candidate recipes, a video generation means for generating cooking procedure videos based on the generated recipes, a user interface means for displaying the generated recipes and cooking procedure videos, an e-commerce means for purchasing additional ingredients, and a means for using prompts when generating candidate recipes. This allows users to efficiently use ingredients in their refrigerators, easily create new dishes, and easily purchase additional ingredients. Furthermore, the cooking procedures can be visually understood through the videos, reducing the risk of making mistakes in cooking procedures.

[1357] The "means for inputting ingredient information" is a means by which a user inputs information about ingredients stored in the refrigerator and transmits that information to the system.

[1358] The "generative AI means" is an artificial intelligence means for analyzing input ingredient information, generating appropriate prompt sentences, and generating cooking recipes based on these.

[1359] The "video generation means" is a means for generating a video including cooking procedures based on the generated cooking recipe.

[1360] The "user interface means" is a means for displaying the generated cooking recipe and cooking procedure video to the user.

[1361] An "electronic commerce means" is a means for purchasing additional ingredients that are needed, and serves to obtain inventory and price information and assist with the purchasing process.

[1362] "Method using prompt sentences" refers to a method of generating a cooking recipe by inputting a prompt sentence created based on ingredient information into a generation AI.

[1363] This invention is a system that proposes cooking recipes using ingredients in a user's refrigerator and generates cooking videos for those recipes. This system allows users to efficiently create new cooking menus. It also allows users to easily purchase additional ingredients as needed to increase the variety of dishes.

[1364] System Overview

[1365] 1. Ingredient input method:

[1366] The user launches the application on their smartphone or PC and manually inputs the ingredients in their refrigerator. This ingredient information is converted to JSON format by the device and sent to the server.

[1367] 2. Generation AI means:

[1368] The server analyzes the received ingredient information and passes the data to the generation AI to generate a cooking recipe. Based on ingredient information such as "chicken, cabbage, and carrots," the generation AI generates multiple recipes using the following example prompt:

[1369] Prompt Sentence Examples

[1370] Please suggest a recipe that can be made using the following ingredients:

[1371] chicken meat

[1372] cabbage

[1373] Carrots

[1374] The generation AI takes the above prompt as input and generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[1375] 3. Video generation method:

[1376] The server then provides the generated recipe information to a video generation AI, which generates a video with cooking instructions in a visually easy-to-understand format. A deep learning-based video generation model, such as PaddleGAN, is used to generate the video. The generated video file is then returned to the server.

[1377] 4. User Interface Means:

[1378] The server sends the acquired recipe information and the generated video to the user's device and displays them through a user interface. The user can view the suggested recipes and videos and select their favorite dishes.

[1379] 5. Electronic commerce instruments:

[1380] If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The delivery status is notified to the user in real time.

[1381] Specific examples

[1382] A specific example will be described in which the user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator.

[1383] 1. User: Opens the smartphone app and manually enters the ingredients in the refrigerator: "chicken," "cabbage," and "carrots."

[1384] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[1385] 3. Server: Analyzes the received ingredient list, inputs "chicken, cabbage, carrots" into the generation AI, and generates the following recipe.

[1386] Stir-fried chicken

[1387] Cabbage and carrot salad

[1388] 4. Server: Requests the video generation AI to generate a cooking video for the above recipe, and returns the generated video file to the server.

[1389] 5. Server: Sends the generated recipes and videos to the user's device.

[1390] 6. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[1391] 7. User: Review suggested recipes and videos and select "Chicken Stir-fry."

[1392] 8. E-commerce method: If the user does not have the necessary additional ingredients (e.g., soy sauce), the user requests to purchase the additional ingredients through the app. The device sends the list of ingredients to the e-commerce site.

[1393] 9. Server: Retrieves inventory and price information from the e-commerce site's API and presents it to the user.

[1394] 10. User: Confirms purchase.

[1395] 11. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[1396] 12. Server: Notifies the user of delivery status in real time.

[1397] 13. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[1398] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, reduces food waste, and improves the user experience.

[1399] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1400] Program processing flow

[1401] Step 1: Enter ingredient information

[1402] User: Launches the application on their smartphone or PC and manually inputs the ingredients they have in their refrigerator. They add ingredients such as "chicken," "cabbage," and "carrots" to the input fields and clicks the submit button.

[1403] Input: Information about ingredients in the refrigerator (e.g., "chicken," "cabbage," "carrots")

[1404] Terminal: Convert the input ingredient information into JSON format.

[1405] Output: JSON formatted ingredient information

[1406] json

[1407] {

[1408] "ingredients": ["chicken", "cabbage", "carrot"]

[1409] }

[1410] What happens: The device collects user-entered information and executes code to convert it into a JSON object, for example, using JavaScript or Python program code.

[1411] Step 2: Submit ingredient information

[1412] Terminal: Sends the converted JSON formatted ingredient information to the server.

[1413] Input: JSON formatted ingredient information

[1414] Output: Ingredient information sent to the server

[1415] What it does: Executes code to create an HTTP POST request and send it to the specified URL on the server.

[1416] Step 3: Recipe generation

[1417] Server: Analyze the received JSON formatted ingredient information.

[1418] Input: JSON formatted ingredient information

[1419] Output: A list of ingredients

[1420] What it does: Uses a JSON parsing library to get the list of ingredients.

[1421] Server: Enters prompt text into the generation AI based on the analyzed data.

[1422] Input: List of ingredients

[1423] Generative AI: Generates cooking recipes based on prompts.

[1424] Prompt Sentence Examples

[1425] Please suggest a recipe that can be made using the following ingredients:

[1426] chicken meat

[1427] cabbage

[1428] Carrots

[1429] Output: Multiple cooking recipes (e.g. "Stir-fried chicken" and "Cabbage and carrot salad")

[1430] Specific operation: Send a prompt to the generation AI (e.g., GPT model) and obtain the generated recipe list.

[1431] Step 4: Save the recipe information

[1432] Server: Stores the generated recipe information in a database.

[1433] Input: Multiple cooking recipes

[1434] Output: Recipe information stored in the database

[1435] What it does: Executes code to add recipe information to the database using an SQL INSERT statement.

[1436] Step 5: Generate cooking videos

[1437] Server: Provides recipe information to the video generation AI.

[1438] Input: Cooking recipe

[1439] Output: Cooking videos generated by video generation AI

[1440] Specific operation: Create an HTTP POST request and send recipe data to the video generation AI API.

[1441] Video generation AI: Generates videos including cooking instructions based on the provided recipe information.

[1442] Input: Cooking recipe

[1443] Output: Cooking videos

[1444] What it does: Runs code that uses a deep learning model to generate a series of image frames and stitches them together into a video.

[1445] Step 6: Submit your recipe and video

[1446] Server: Sends the generated recipe information and cooking videos to the user's device.

[1447] Input: Cooking recipes, cooking videos

[1448] Output: Recipe information and video sent to the user's device

[1449] Specific behavior: Executes code to send data to the user's device via an HTTP GET or POST request.

[1450] Step 7: Display in the user interface

[1451] Terminal: Parses the received data and displays it in the application's user interface.

[1452] Input: Cooking recipes, cooking videos

[1453] Output: Recipe information and video displayed in the user interface

[1454] Specific operation: Analyzes the received JSON data, displays the recipe title, ingredients, and cooking steps in text format, and simultaneously executes code to play the cooking video on a video player.

[1455] Step 8: Purchase additional ingredients

[1456] User: If an additional ingredient (e.g. soy sauce) is needed, the user wishes to purchase the additional ingredient in the app. Click the purchase button and select the required ingredient.

[1457] Terminal: Send the list of ingredients you wish to purchase in JSON format to the e-commerce site.

[1458] Input: List of ingredients you wish to purchase

[1459] Output: Ingredient list sent to the e-commerce site

[1460] What it does: Executes code to create an HTTP POST request and send it to the e-commerce site's API.

[1461] Step 9: Get inventory and pricing information

[1462] Server: Obtains inventory and price information from the e-commerce site's API.

[1463] Input: List of ingredients you wish to purchase

[1464] Output: Stock and price information obtained from the e-commerce site

[1465] Specific behavior: Executes code to retrieve information from the e-commerce site's API via an HTTP GET request and receive it in JSON format.

[1466] Server: Presents stock and price information to the user. Sends data to the device and displays it on the user's screen.

[1467] Input: Stock and price information obtained from the e-commerce site

[1468] Output: Stock and pricing information displayed to the user.

[1469] What it does: Sends data to the device and executes code to display it in the user interface.

[1470] Step 10: Confirm purchase and process order

[1471] User: Confirms purchase by clicking the Confirm button on the screen.

[1472] Input: Purchase confirmation instructions

[1473] Output: Confirmed information

[1474] What it does: Receives input from the user interface and executes code to send purchase confirmation information to the server.

[1475] Server: Receives purchase confirmation information, sends it to the EC site's API, confirms the order, and begins the shipping process.

[1476] Input: Purchase confirmation information

[1477] Output: Order information for the e-commerce site

[1478] What it does: Executes code to create an HTTP POST request and send it to the e-commerce site's API.

[1479] Step 11: Delivery Status Notification

[1480] Server: Notify the user of the delivery status in real time, for example via push notifications or in-app messages.

[1481] Input: Delivery status information

[1482] Output: User notification

[1483] What it does: Executes code to notify the user of delivery status using a push notification server or an in-app notification system.

[1484] As described above, this system efficiently utilizes ingredients in the user's refrigerator and provides new cooking recipes and cooking videos, thereby increasing the variety of dishes available and simplifying the purchase of additional ingredients.

[1485] (Application example 1)

[1486] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1487] Conventional recipe suggestion systems offer limited options for efficiently preparing dishes using ingredients users already have, and lack content that allows users to visually understand cooking methods. This makes it time-consuming and labor-intensive for users to create new recipes, making it difficult to efficiently utilize ingredients. Purchasing additional ingredients is also time-consuming and often burdensome for users. This limits the variety of dishes available and leads to problems such as wasted ingredients.

[1488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1489] In this invention, the server includes a means for inputting food information, a generative AI model means for analyzing the input food information and generating candidate recipes, a video generation means for generating cooking procedure videos based on the generated recipes, a user interface means for displaying the generated recipes and cooking procedure videos, and an online commerce means for purchasing additional ingredients. This allows users to efficiently use ingredients in their refrigerators and easily create new recipes. Furthermore, by making cooking procedures easier to understand visually through videos, cooking time is reduced and additional ingredients can be purchased more smoothly. This reduces food waste and improves the user experience.

[1490] The "means for inputting food information" is an interface that allows a user to input information about ingredients and other food items in the refrigerator using a terminal that the user owns and transmits this information to the system.

[1491] The "generative AI model means" is a module that uses artificial intelligence technology to generate optimal cooking methods based on input food information.

[1492] The "video generation means" is a technology for generating a visual video of cooking procedures based on the generated recipes, so that users can easily understand how to cook.

[1493] The "user interface means" is an interface that displays the generated recipe and cooking procedure videos to the user and allows the user to view and operate them.

[1494] "Online commerce means" refers to a means by which a user can make an online transaction to purchase additional ingredients that the user needs.

[1495] This system proposes recipes using ingredients that the user has on hand, and generates and distributes videos of the cooking steps, allowing users to efficiently create new recipes and easily purchase any additional ingredients they need.

[1496] System Overview

[1497] The system includes the following means:

[1498] 1. Means for entering food information:

[1499] The user inputs the ingredients in the refrigerator using a device such as a smartphone. This ingredient information is converted into JSON format by the device and sent to the server.

[1500] 2. Generative AI model means:

[1501] The server analyzes the received ingredient information and passes the data to a generative AI to generate recipes. For example, based on ingredient information such as "chicken, cabbage, and carrots," the generative AI model generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[1502] 3. Video generation method:

[1503] The server provides the generated recipe information to the video generation AI, which generates a video with cooking instructions in a visually easy-to-understand format. The generated video file is then returned to the server.

[1504] 4. User Interface Means:

[1505] The server sends the acquired recipe information and the generated video to the user's device and displays them through a user interface. The user can view the suggested recipes and videos and select their favorite dishes.

[1506] 5. Online commerce methods:

[1507] If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The delivery status is notified to the user in real time.

[1508] Program processing

[1509] The system uses the following hardware and software:

[1510] Hardware

[1511] Smartphones and tablets (user devices)

[1512] Cloud Server

[1513] software

[1514] On the server side, recipe generation AI (e.g., GPT-4) and video generation AI (e.g., DALL-E and Deep Video) are used.

[1515] On the client side, a communication library (e.g., requests) is used to send and receive data in JSON format.

[1516] A standard web front-end framework (e.g., React, Angular, etc.) is used to display the user interface.

[1517] Specific examples

[1518] As a specific example, consider the case where the user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator.

[1519] Prompt Sentence Examples

[1520] Suggest a simple and delicious recipe using the following ingredients you have in your fridge:

[1521] chicken meat

[1522] cabbage

[1523] Carrots

[1524] The server passes this prompt to the generation AI, which generates a recipe like this:

[1525] "Stir-fried chicken" and "Cabbage and carrot salad"

[1526] The server then passes this recipe information to a video generation AI, which generates a video of the cooking steps. The generated data is stored in cloud data storage and sent to the user's device. Users can view these recipes and videos through a smartphone app and begin cooking based on the recipe they select. If additional ingredients are required, they can easily purchase them through the e-commerce site.

[1527] This system allows users to make the most of the ingredients they have, reduce waste, and easily enjoy new cooking menus.

[1528] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1529] Step 1:

[1530] The user launches the app on their smartphone and inputs the ingredients they have in their refrigerator. The user enters the specific ingredients they have (e.g., chicken, cabbage, carrots) into the app. This information is converted into a data format (e.g., JSON) and sent to the server.

[1531] Input: Ingredient information entered by the user

[1532] Output: Ingredient information in JSON format

[1533] Step 2:

[1534] The server analyzes the received ingredient information and passes the data to the generative AI model. The server then generates a prompt based on the ingredient information and inputs it into the generative AI model (e.g., GPT-4). During this analysis, the input ingredient information is formatted appropriately.

[1535] Input: JSON formatted ingredient information

[1536] Output: A prompt to be passed to the generative AI model

[1537] Step 3:

[1538] The server uses a generative AI model to generate multiple recipes based on the input prompt, which analyzes and generates several candidate recipes based on the user's ingredients.

[1539] Input: prompt statement

[1540] Output: Multiple generated recipes

[1541] Step 4:

[1542] The server provides the generated recipe information to the video generation AI, which then generates cooking instructions as videos. The video generation AI then generates video files that visually explain the steps of each recipe in an easy-to-understand format.

[1543] Input: Generated recipe information

[1544] Output: Video file visualizing the cooking steps

[1545] Step 5:

[1546] The server sends the generated recipes and videos to the user's device, stores them in cloud storage, and delivers them to the user's app in real time.

[1547] Input: Generated recipes and videos

[1548] Output: User device where recipes and videos are delivered

[1549] Step 6:

[1550] Users can browse suggested recipes and videos through the app and select their preferred dish. Users can then select one of the multiple options on the app and see how to cook it.

[1551] Input: Distributed recipes and videos

[1552] Output: User selected recipe

[1553] Step 7:

[1554] If the user wishes to purchase additional ingredients, the device sends a list of the ingredients they wish to purchase to the EC site. The device then creates a list of the ingredients needed based on the selected recipe and sends the data to the EC site.

[1555] Input: A list of additional ingredients based on the recipe selected by the user

[1556] Output: Ingredient list sent to the e-commerce site

[1557] Step 8:

[1558] The server calls the API of the e-commerce site, obtains stock and price information for each ingredient, and presents it to the user. The server analyzes the API response and presents the information in a format that is easy for the user to understand.

[1559] Input: List of additional ingredients sent

[1560] Output: Retrieved inventory and price information

[1561] Step 9:

[1562] The user confirms the purchase procedure through the app and confirms the purchase. The app receives the user's input and sends it to the server.

[1563] Input: User's purchase confirmation information

[1564] Output: Purchase confirmation information

[1565] Step 10:

[1566] The server sends the purchase confirmation information to the e-commerce site's API to complete the order process. The server also sends the order information via the API to start the logistics process.

[1567] Input: User's purchase confirmation information

[1568] Output: Order information sent to the e-commerce site

[1569] Step 11:

[1570] The server notifies the user of the delivery status in real time. The server obtains the delivery status from the EC site and notifies the user.

[1571] Input: Delivery status from the e-commerce site

[1572] Output: Real-time notification of delivery status

[1573] This process allows users to effectively use the ingredients they have and smoothly purchase any additional ingredients they need. The generated cooking video also reduces the time and effort required for cooking.

[1574] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1575] The present invention is a system that suggests cooking recipes using ingredients in the user's refrigerator and generates cooking videos for those recipes. By further combining this system with an emotion engine that recognizes the user's emotions, it is possible to suggest recipes and videos that correspond to the user's emotional state. This system not only enables users to efficiently create new cooking menus, but also provides a more satisfying experience by receiving suggestions that match their emotions. Specific program processing for this system is described below.

[1576] System Overview

[1577] 1. Ingredient input method: The user launches the application on their smartphone or PC and inputs the ingredients they have in their refrigerator. This ingredient information is converted to JSON format by the device and sent to the server.

[1578] 2. Recipe generation: The server analyzes the received ingredient information and passes the data to the generation AI to generate a cooking recipe. For example, based on ingredient information such as "chicken, cabbage, and carrots," it generates recipes such as "stir-fried chicken" or "cabbage and carrot salad."

[1579] 3. Emotion Engine: When a user is using the application, sensors such as a camera and microphone are used to analyze the user's facial expressions and voice to recognize their emotional state. This emotional information is then sent to the server.

[1580] 4. Emotion-based suggestions: The server analyzes the emotion information received from the emotion engine and adjusts the recipe suggestions based on the user's current emotional state. For example, if the user is feeling stressed, it will suggest dishes with a relaxing effect.

[1581] 5. Video generation method: The server provides recipe information and emotion information to the video generation AI, which generates a video with cooking instructions that match the emotion and is easy to understand visually. The generated video file is returned to the server.

[1582] 6. User interface means: The server sends the acquired recipe information, emotion information, and generated video to the user's device and displays them through the user interface. The user can view the suggested recipes and videos and choose their favorite dishes.

[1583] 7. E-commerce method: If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The delivery status is notified to the user in real time.

[1584] Specific examples

[1585] As a specific example, consider a case where a user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator, and the emotion engine recognizes that the user is feeling stressed.

[1586] 1. User: Opens the app on his smartphone and enters the ingredients in his refrigerator: "chicken," "cabbage," and "carrots."

[1587] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[1588] 3. Server: Analyzes the received ingredient list, inputs "chicken, cabbage, carrots" into the generation AI, and generates the following recipes ("stir-fried chicken" and "cabbage and carrot salad").

[1589] 4. Emotion engine: Analyzes the user's facial expressions and voice and notifies the server that the user is feeling stressed.

[1590] 5. Server: Analyzes information from the emotion engine and prioritizes recipes that have a relaxing effect.

[1591] 6. Video generation method: The video generation AI is requested to generate a cooking video for the "cabbage and carrot salad" recipe, which is particularly relaxing from the above recipes. The generated video file is returned to the server.

[1592] 7. Server: Sends the generated recipes and videos to the user's device.

[1593] 8. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[1594] 9. User: Review suggested relaxing recipes and videos and choose to cook.

[1595] 10. E-commerce method: If the user does not have the necessary additional ingredients (e.g., olive oil), the user requests to purchase the additional ingredients through the app. The device sends the list of ingredients to the e-commerce site.

[1596] 11. Server: Retrieves olive oil inventory and price information from the e-commerce site's API and presents it to the user.

[1597] 12. User: Confirms purchase.

[1598] 13. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[1599] 14. E-commerce: Process orders and initiate logistics processes in a short time.

[1600] 15. Server: Notifies the user of delivery status in real time.

[1601] 16. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[1602] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, and provides a more satisfying cooking experience by making suggestions based on the user's emotional state.

[1603] The processing flow will be explained below.

[1604] Step 1:

[1605] User: Launches the application on their smartphone or PC and accesses the screen where they can input the ingredients in their refrigerator.

[1606] Step 2:

[1607] User: Enter the ingredients they currently have on hand, such as "chicken," "cabbage," and "carrots," into the text field.

[1608] Step 3:

[1609] Terminal: Convert the input ingredient list into JSON format and send it to the server.

[1610] Step 4:

[1611] Server: Analyzes the received ingredient list and converts it into a format that can be passed as input data to the generation AI.

[1612] Step 5:

[1613] Server: Pass the ingredients list to the generation AI and have it generate candidate cooking recipe information.

[1614] Step 6:

[1615] Generative AI: Based on a list of ingredients, it generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[1616] Step 7:

[1617] Server: Receives the generated recipe information.

[1618] Step 8:

[1619] Emotion engine: When a user is using an application, the system captures the user's facial expressions and voice data via the camera and microphone, and analyzes this to recognize the user's emotional state.

[1620] Step 9:

[1621] Emotion engine: Sends the recognized emotional state to the server. For example, if the user is feeling stressed, it sends the data of the stress state.

[1622] Step 10:

[1623] Server: Analyzes the emotional information received from the emotion engine and adjusts recipe suggestions based on the user's current emotional state, for example, prioritizing dishes with a relaxing effect.

[1624] Step 11:

[1625] Server: Provides recipe information and emotional information to the video generation AI and asks it to generate a cooking instruction video that matches the emotion.

[1626] Step 12:

[1627] Video generation AI: Based on recipe information, it simulates cooking steps and generates visually easy-to-understand cooking videos.

[1628] Step 13:

[1629] Video generation AI: Returns the generated video file to the server.

[1630] Step 14:

[1631] Server: Sends the recipe information and generated video together to the user's device.

[1632] Step 15:

[1633] Terminal: Analyzes the recipe information and video sent from the server and formats them for display on the user interface.

[1634] Step 16:

[1635] Device: The user interface displays recipes such as "Stir-fried chicken" and "Cabbage and carrot salad," along with cooking videos for each recipe that can be played.

[1636] Step 17:

[1637] User: Review suggested recipes and videos and select the dish they prefer.

[1638] Step 18:

[1639] User: Sees an ingredient needed for a selected recipe that is not in the refrigerator (e.g., soy sauce) and asks to purchase it within the app.

[1640] Step 19:

[1641] Terminal: Convert the list of ingredients desired for purchase into JSON format and send it to the EC site's API.

[1642] Step 20:

[1643] Server: Obtains food inventory and price information from the EC site's API and presents it to the user.

[1644] Step 21:

[1645] User: Confirms purchase.

[1646] Step 22:

[1647] Terminal: Generates a request to send purchase confirmation information to the e-commerce site's API via the server.

[1648] Step 23:

[1649] Server: Sends the received request to the e-commerce site and confirms the order.

[1650] Step 24:

[1651] E-commerce sites: process orders and initiate logistics processes quickly.

[1652] Step 25:

[1653] Server: Receives real-time delivery status information from the e-commerce site and notifies the user.

[1654] Step 26:

[1655] User: Check delivery status and wait for additional ingredients to arrive.

[1656] Step 27:

[1657] User: When additional ingredients arrive, check their arrival on the device and proceed with cooking preparations.

[1658] Step 28:

[1659] User: Starts cooking the selected dish using the provided recipe and video.

[1660] The above are the specific processing steps of the system according to the present invention. This process not only allows users to efficiently use the ingredients in their refrigerator to enjoy a variety of dishes, but also provides a more satisfying cooking experience by receiving recipe suggestions that match their emotions.

[1661] Example 2

[1662] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1663] Conventional recipe suggestion systems are unable to consider the user's emotional state when proposing recipes, making it difficult to enhance the user's psychological satisfaction. Furthermore, they lack a means to provide easy-to-understand visual instructions for cooking, making them difficult to use, especially for beginners. Another issue is the complicated process of purchasing additional ingredients, which makes it difficult to start cooking smoothly.

[1664] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1665] In this invention, the server includes a means for inputting ingredient information, an artificial intelligence means for analyzing the input ingredient information and generating candidate recipes, an emotion engine means for recognizing the user's emotional state and adjusting the cooking recipe based on the recognized emotional information, a means for generating cooking procedure videos based on the generated cooking recipe, a user interface means for displaying the generated cooking recipe and cooking procedure videos, and an electronic commerce means for purchasing any additional ingredients required. This allows the user to receive recipe suggestions suited to their emotional state and to proceed with cooking while referring to the visually easy-to-understand cooking videos. It also makes it easy to purchase any additional ingredients required.

[1666] The "means for inputting ingredient information" is an interface that allows users to input ingredients in the refrigerator as text or images via a smartphone or PC application.

[1667] The "artificial intelligence means" is a system that analyzes input ingredient information and generates candidate cooking recipes using a generative AI model.

[1668] The "emotion engine means" is a function that analyzes the user's facial expressions and voice, recognizes the user's emotional state, and adjusts the cooking recipe based on the emotional information.

[1669] The "means for generating cooking procedure videos" is a technology for generating cooking procedure videos that are visually easy to understand based on the generated cooking recipe.

[1670] The "user interface means" is an interface for displaying the generated cooking recipe and cooking procedure video to the user.

[1671] "Electronic commerce means" refers to a system that provides online shopping functionality for purchasing additional ingredients needed.

[1672] The present invention is a system that suggests cooking recipes using ingredients in the user's refrigerator and generates cooking videos for those recipes. Furthermore, by combining it with an emotion engine that recognizes the user's emotional state, it suggests recipes and videos according to that emotional state. This system allows users to efficiently create new cooking menus, and by receiving suggestions that match their emotions, it increases their satisfaction. The system's program processing is explained in detail below.

[1673] System Overview

[1674] Entering ingredient information

[1675] Users enter information about ingredients in their refrigerators through a smartphone or PC application. Input methods include text entry and uploading images taken with a camera. The entered information is converted to JSON format by the device and sent to the server.

[1676] Recipe Generation

[1677] The server analyzes the received JSON-formatted ingredient information and sends a prompt to the generative AI model (e.g., GPT-3) saying, "Please suggest a cooking recipe that includes chicken, cabbage, and carrots." The generative AI model generates multiple recipes based on the received prompt.

[1678] emotion recognition

[1679] When a user is using an application, sensors such as a camera and microphone are used to record facial expressions and voice. The device then uses the recorded data to analyze the user's emotional state using an emotion recognition engine (e.g., Microsoft Azure Emotion API). The analyzed emotional information is converted into JSON format and sent to the server.

[1680] Emotion-based recipe suggestions

[1681] The server analyzes the received emotional information and adjusts the recipe suggestions based on the user's current emotional state. For example, if the user is feeling stressed, it will prioritize suggestions of cooking recipes that have a relaxing effect. As a specific example, it generates "stir-fried chicken" and "cabbage and carrot salad," and prioritizes displaying the "cabbage and carrot salad," which has the effect of relieving stress.

[1682] Cooking video generation

[1683] The server requests a video generation AI (e.g., Synthesia) to generate a video explaining the cooking steps for the specified recipe based on the emotional state and recipe information. The generated video file is sent back to the server and provided to the user.

[1684] User display and operation

[1685] The server sends the acquired recipe information, emotion information, and generated cooking videos to the user's device, which analyzes the information and displays it through a user interface. The user can then check and operate the suggested recipes and videos.

[1686] Purchase additional ingredients

[1687] If the user does not have the necessary ingredients to prepare a suggested recipe, they can request to purchase the ingredients through the app. The device sends a list of the ingredients they wish to purchase to the EC site via an e-commerce transaction method. The server retrieves inventory and price information from the EC site's API and presents it to the user. Once the user confirms the purchase, the server sends the order information to the EC site's API and confirms the order. The user is notified of the delivery status in real time.

[1688] Specific examples

[1689] As a specific example, consider a case where a user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator, and the emotion engine recognizes that the user is feeling stressed.

[1690] 1. User: Opens the app on his smartphone and enters the ingredients in his refrigerator: "chicken," "cabbage," and "carrots."

[1691] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[1692] 3. Server: Analyzes the received list of ingredients, inputs "chicken, cabbage, carrots" into a generation AI (e.g., GPT-3), and generates recipes such as the following ("stir-fried chicken" and "cabbage and carrot salad").

[1693] 4. Emotion engine: Analyzes the user's facial expressions and voice and notifies the server that the user is feeling stressed.

[1694] 5. Server: Analyzes information from the emotion engine and prioritizes recipes that have a relaxing effect.

[1695] 6. Video generation method: A video generation AI (e.g., Synthesia) is requested to generate a cooking video for the "cabbage and carrot salad" recipe, which is particularly relaxing from the above recipes. The generated video file is returned to the server.

[1696] 7. Server: Sends the generated recipes and videos to the user's device.

[1697] 8. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[1698] 9. User: Review suggested relaxing recipes and videos and choose to cook.

[1699] 10. E-commerce method: If you don't have the additional ingredients you need (e.g. olive oil), submit a list of ingredients you want to purchase to the e-commerce site.

[1700] 11. Server: Retrieves olive oil inventory and price information from the e-commerce site's API and presents it to the user.

[1701] 12. User: Confirms purchase.

[1702] 13. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[1703] 14. E-commerce website: Processes orders and initiates the delivery process.

[1704] 15. Server: Notifies the user of delivery status in real time.

[1705] 16. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[1706] Prompt Sentence Examples

[1707] An example of a prompt sentence to input into the generative AI model is, "Please suggest a relaxing cooking recipe that includes chicken, cabbage, and carrots."

[1708] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, and provides a more satisfying cooking experience by making suggestions based on the user's emotional state.

[1709] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1710] Step 1:

[1711] The user enters ingredient information

[1712] Users launch the application on their smartphone or PC and enter information about the ingredients in their refrigerator. Specifically, they enter "chicken," "cabbage," and "carrots" in the text input field and press the send button. Alternatively, they can take a photo of the ingredients and upload it.

[1713] input:

[1714] Ingredient information (text or image)

[1715] output:

[1716] Ingredient information is saved on the device

[1717] Step 2:

[1718] The device analyzes the ingredient information and sends it to the server.

[1719] The device extracts the names of ingredients from the input text data as is, or from image data using an image recognition engine (e.g., Google Cloud Vision API). The extracted names are converted to JSON format and sent to the server.

[1720] input:

[1721] Ingredient information (text or image)

[1722] Data processing / calculation:

[1723] Convert text information (directly into JSON)

[1724] Recognition processing using an image analysis engine (image → text information)

[1725] output:

[1726] JSON formatted ingredient information

[1727] Step 3:

[1728] The server analyzes the sent ingredient information and sends a prompt to the generative AI model.

[1729] The server analyzes the received JSON-formatted ingredient information and sends a prompt to the generative AI model (e.g., GPT-3) such as, "Please suggest a cooking recipe that includes chicken, cabbage, and carrots."

[1730] input:

[1731] JSON formatted ingredient information

[1732] Data processing / calculation:

[1733] Prompt statement generation and sending (creating a prompt statement and sending it)

[1734] output:

[1735] Prompt to send to the generative AI model

[1736] Step 4:

[1737] A generative AI model generates cooking recipes based on prompts

[1738] The generative AI model receives a prompt and generates potential recipes based on the prompt, such as "stir-fried chicken" or "cabbage and carrot salad."

[1739] input:

[1740] Prompt statement

[1741] Data Calculation:

[1742] Recipe generation using natural language processing

[1743] output:

[1744] Cooking recipe information

[1745] Step 5:

[1746] The server receives the generated recipe and obtains the emotion information.

[1747] The server stores the generated recipe information and then prepares to acquire the user's emotional information, specifically, issues an instruction to the terminal to acquire the user's current emotional state.

[1748] input:

[1749] Cooking recipe information

[1750] output:

[1751] Instructions for obtaining emotional information

[1752] Step 6:

[1753] The device records the user's facial expressions and voice and analyzes them using an emotion engine.

[1754] When a user is using an application, the device's camera and microphone are used to record facial expressions and voice. The device then uses the recorded data to analyze the user's emotional state using an emotion recognition engine (e.g., Microsoft Azure Emotion API).

[1755] input:

[1756] Facial expression data, voice data

[1757] Data processing / calculation:

[1758] Analysis by emotion recognition engine

[1759] output:

[1760] Emotion information in JSON format

[1761] Step 7:

[1762] The device sends the analyzed emotional information to the server.

[1763] The device converts the analysis results into emotional information in JSON format and sends it to the server. For example, information such as "Emotion: Stress" is sent.

[1764] input:

[1765] Emotion information (JSON format)

[1766] output:

[1767] Send emotion information to the server

[1768] Step 8:

[1769] The server analyzes the emotional information and adjusts the recipe suggestions.

[1770] The server analyzes the received emotional information and adjusts recipe suggestions based on the user's current emotional state. For example, if a user is feeling stressed, it will prioritize recipes that have a relaxing effect.

[1771] input:

[1772] Emotional information, cooking recipe information

[1773] Data processing / calculation:

[1774] Recipe adjustment based on emotion information

[1775] output:

[1776] Adjusted cooking recipe information

[1777] Step 9:

[1778] The server requests the video generation AI to generate a cooking procedure video.

[1779] Based on the emotional state and recipe information, the server requests a video generation AI (e.g., Synthesia) to generate a video explaining the cooking steps for the specified recipe.

[1780] input:

[1781] Adjusted cooking recipe information

[1782] Data Calculation:

[1783] Video generation by video generation AI

[1784] output:

[1785] Cooking procedure video

[1786] Step 10:

[1787] The server sends the generated recipe and video to the device.

[1788] The server sends the generated recipe information and cooking procedure video to the user's terminal.

[1789] input:

[1790] Cooking procedure videos, adjusted cooking recipe information

[1791] output:

[1792] Send recipe information and videos to your device

[1793] Step 11:

[1794] The device analyzes the received information and displays it on the user interface.

[1795] The device analyzes the recipe information and cooking video sent from the server and displays them on the user interface. The user can then check the suggested recipes and videos and choose their preferred dish.

[1796] input:

[1797] Recipe information, cooking procedure videos

[1798] output:

[1799] Display in the user interface

[1800] Step 12:

[1801] The user performs an operation to purchase any additional ingredients needed.

[1802] If the user does not have the necessary ingredients to prepare the suggested recipe, they can request to purchase the ingredients through the app. They can create a list of ingredients they wish to purchase and send it to their device.

[1803] input:

[1804] Missing ingredient information

[1805] output:

[1806] List of ingredients you want to buy

[1807] Step 13:

[1808] The device sends the list of ingredients desired to be purchased to the e-commerce site and displays the information.

[1809] The terminal sends the list of ingredients desired to be purchased to the EC site via the electronic commerce means. The server retrieves inventory and price information from the EC site's API and presents it to the user.

[1810] input:

[1811] List of ingredients you want to buy

[1812] output:

[1813] Send to EC site and display stock and price information

[1814] Step 14:

[1815] The user confirms the purchase and the server sends the order information to the e-commerce site.

[1816] The user checks the inventory and price information provided and confirms the purchase. The server sends the purchase confirmation information to the EC site's API and confirms the order. The EC site processes the order and starts the delivery process.

[1817] input:

[1818] Purchase confirmation information

[1819] output:

[1820] Send order information to the EC site

[1821] Step 15:

[1822] The server notifies the user of the delivery status in real time.

[1823] The server notifies the user of the delivery status in real time. When the additional ingredients arrive, the user can check the arrival and start cooking. The user can cook according to the recipe while watching the provided video.

[1824] input:

[1825] Delivery Status Information

[1826] output:

[1827] Notify users of delivery status

[1828] (Application example 2)

[1829] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1830] While conventional cooking recommendation systems can suggest recipes based on the ingredients in the refrigerator, they are unable to suggest optimal dishes based on the user's emotional state. This makes it difficult to accurately suggest dishes desired when the user is in a specific emotional state, and there are a lack of means to improve the user experience. Furthermore, there is a lack of functionality to immediately order the suggested dishes and track their delivery status.

[1831] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting ingredient information, artificial intelligence means for analyzing the input ingredient information and generating candidate recipes, means for generating cooking procedure videos based on the generated recipes, user interface means for displaying the generated recipes and cooking procedure videos, e-commerce means for purchasing additional ingredients, an emotion engine for recognizing the user's emotions, means for suggesting recipes based on the user's emotional state, means for ordering the suggested dishes from an external service, and means for tracking the delivery status of the ordered dishes. This enables the server to suggest optimal dishes based on the user's emotional state, and to quickly order the suggested dishes and track their delivery status.

[1832] The "means for inputting ingredient information" refers to the interface or terminal function that allows the user to input ingredient information about ingredients stored in the refrigerator.

[1833] The "artificial intelligence means for analyzing input ingredient information and generating candidate cooking recipes" refers to an artificial intelligence algorithm and calculation mechanism for generating appropriate cooking recipes based on the ingredient information input by the user.

[1834] "Means for generating videos of cooking steps based on generated cooking recipes" refers to technology and devices for creating videos of cooking steps that are visually easy to understand based on cooking recipes generated by AI.

[1835] The "user interface means for displaying the generated cooking recipe and cooking procedure video" refers to an interface and device that allows the user to visually check the generated cooking recipe and cooking procedure video.

[1836] "Electronic commerce means for purchasing additional ingredients" refers to a system and procedure for purchasing ingredients that are in short supply via the Internet.

[1837] An "emotion engine that recognizes user emotions" is a combination of software and hardware that uses a camera and microphone to analyze emotions from the user's facial expressions and voice and recognize those emotions.

[1838] The "means for suggesting recipes according to the user's emotional state" refers to an algorithm and means for selecting and suggesting optimal cooking recipes based on the user's recognized emotions.

[1839] "Means for ordering suggested dishes from an external service" refers to a system and function for ordering the dishes selected by the user from an external food delivery service or the like.

[1840] The "means for tracking the delivery status of an ordered meal" refers to a system and technology for tracking the current delivery status of an ordered meal in real time and notifying the user.

[1841] The present invention is a system that suggests optimal dishes based on a user's emotional state and orders them through an external food delivery service. This system allows users to easily order dishes that match their emotions and track delivery status in real time. Specific embodiments of this system are described below.

[1842] System Overview

[1843] The system of the present invention comprises the following components:

[1844] 1. Ingredient input method:

[1845] Users can input information about ingredients in their refrigerators using an application on their smartphone or PC. Similar applications include smart fridge management apps.

[1846] 2. Artificial Intelligence Means:

[1847] The system analyzes the input ingredient information and generates candidate recipes. This analysis is performed using a generative AI model. For example, if "chicken, cabbage, carrots" is input, recipes such as "stir-fried chicken" and "cabbage and carrot salad" will be generated.

[1848] 3. Video generation method:

[1849] Based on the generated recipe, a video showing the cooking steps is generated. This video generation is also done using an AI model. An intuitive and easy-to-understand video is created for each recipe.

[1850] 4. User Interface Means:

[1851] The generated recipes and cooking video are displayed on the user's device interface, allowing the user to select which dish to cook based on this information.

[1852] 5. Electronic commerce instruments:

[1853] It provides a means for users to purchase additional ingredients they need. When users input the ingredients they are missing, it connects with the API of the e-commerce site to carry out the online purchase process based on that information.

[1854] 6. Emotion Engine:

[1855] The system analyzes the user's facial expressions and voice to recognize their emotional state. This analysis is performed using a camera and microphone, and emotion analysis software such as EmotionEngine is used.

[1856] 7. Emotion-based recipe suggestion method:

[1857] Based on the emotional information obtained from the emotion engine, the system suggests recipes that are optimal for the user's current emotional state. For example, if the user is feeling stressed, the system suggests dishes that have a relaxing effect.

[1858] 8. How to order from third-party services:

[1859] Order the suggested dishes from an external food delivery service, specifically by placing an order with the restaurant via an API such as FoodDeliveryService.

[1860] 9. Delivery status tracking:

[1861] Track your order delivery in real time and get notified every time the delivery status changes.

[1862] Program processing overview

[1863] The server includes a means for inputting ingredient information, an artificial intelligence means for analyzing the input ingredient information and generating candidate recipes, and a means for generating videos of cooking procedures based on the generated recipes, thereby enabling the server to suggest optimal dishes according to the user's emotional state, and to quickly order the dishes and track their delivery status.

[1864] Specific examples

[1865] For example, if a user inputs that they have "chicken," "cabbage," and "carrots" in their refrigerator, and the app analyzes their facial expressions and voice and determines that they are feeling stressed, it will suggest a relaxing "cabbage and carrot salad." When the user selects that recipe, it places an order through an external food delivery service and tracks the progress of the ingredients in real time until they are delivered.

[1866] Prompt Sentence Examples

[1867] "A system that suggests the best meal for a stressed user who has chicken, cabbage, and carrots in the refrigerator, orders the meal, and tracks its delivery."

[1868] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1869] Step 1:

[1870] The user launches the application on their smartphone or PC and enters information about ingredients in the refrigerator. The ingredient information entered by the user is converted to JSON format data by the device and sent to the server. This generates (input: ingredient information entered by the user) (output: ingredient data in JSON format).

[1871] Step 2:

[1872] The server analyzes the received JSON-formatted ingredient information and generates a cooking recipe based on the ingredient information. This process uses a generative AI model. For example, if the data "chicken, cabbage, carrots" is input, the server generates candidate recipes such as "stir-fried chicken" or "cabbage and carrot salad." (Input: JSON-formatted ingredient data) (Output: candidate cooking recipe) is generated.

[1873] Step 3:

[1874] When a user is operating an application, the device's camera and microphone are used to detect the user's facial expressions and voice, which are then analyzed by the emotion engine. The analysis results (e.g., information on the user's stress level) are sent to the server, where (input: user's facial expression data and voice data) (output: emotion analysis results) are generated.

[1875] Step 4:

[1876] The server uses the emotion analysis results received from the emotion engine to prioritize recipe suggestions that match the user's emotional state. In this case, recipes that are expected to have a relaxing effect or improve mood are selected. (Input: Candidate recipes and emotion analysis results) (Output: Optimal recipe suggestions based on emotions) are generated.

[1877] Step 5:

[1878] The server generates a cooking procedure video based on the generated recipe using a video generation means. The generated cooking procedure video is returned to the server. (Input: optimal cooking recipe) (Output: cooking procedure video) is generated.

[1879] Step 6:

[1880] The server sends the generated recipes and cooking instruction videos to the user's device. The user can check these suggestions in the application and understand them visually. (Input: Cooking recipes and cooking instruction videos) (Output: Recipe and video data sent to the user's device) is generated.

[1881] Step 7:

[1882] When the user selects the best recipe from the suggestions, the device sends the order data to an external food delivery service. For example, it orders the food from a nearby restaurant via FoodDeliveryService. (Input: Selected recipe) (Output: Order data to external service) is generated.

[1883] Step 8:

[1884] The server tracks the delivery status of the order in real time via the food delivery service's API. Every time the delivery status information is updated, the server sends it to the terminal and notifies the user. (Input: order data and delivery status information) (Output: real-time delivery status notification) is generated.

[1885] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1886] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1887] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1888] [Fourth embodiment]

[1889] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1890] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1891] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1892] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1893] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1894] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1895] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1896] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1897] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1898] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1899] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1900] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1901] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1902] The present invention is a system that proposes cooking recipes using ingredients in a user's refrigerator and generates cooking videos for those recipes. This system allows users to efficiently create new cooking menus. It also allows users to easily purchase additional ingredients as needed to increase the variety of dishes. The specific program processing of this system is described below.

[1903] System Overview

[1904] 1. Ingredient input method: The user launches the application on their smartphone or PC and inputs the ingredients they have in their refrigerator. This ingredient information is converted to JSON format by the device and sent to the server.

[1905] 2. Recipe generation: The server analyzes the received ingredient information and passes the data to the generation AI to generate a cooking recipe. For example, based on ingredient information such as "chicken, cabbage, and carrots," the generation AI generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[1906] 3. Video generation means: The server provides the generated recipe information to the video generation AI, which generates a video showing the cooking steps in a visually easy-to-understand format. The generated video file is returned to the server.

[1907] 4. User interface: The server sends the acquired recipe information and the generated video to the user's device and displays them through the user interface. The user can view the suggested recipes and videos and select their favorite dishes.

[1908] 5. E-commerce method: If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The user is notified of the delivery status in real time.

[1909] Specific examples

[1910] As a specific example, consider the case where a user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator.

[1911] 1. User: Opens the app on his smartphone and enters the ingredients in his refrigerator: "chicken," "cabbage," and "carrots."

[1912] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[1913] 3. Server: Analyzes the received ingredient list, inputs "chicken, cabbage, carrots" into the generation AI, and generates the following recipes ("stir-fried chicken" and "cabbage and carrot salad").

[1914] 4. Video generation method: Request the video generation AI to generate a cooking video for the above recipe. The generated video file is returned to the server.

[1915] 5. Server: Sends the generated recipes and videos to the user's device.

[1916] 6. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[1917] 7. User: Review suggested recipes and videos and select "Chicken Stir-fry."

[1918] 8. E-commerce method: If the user does not have the necessary additional ingredients (e.g., soy sauce), the user requests to purchase the additional ingredients through the app. The device sends the list of ingredients to the e-commerce site.

[1919] 9. Server: Retrieves soy sauce inventory and price information from the e-commerce site's API and presents it to the user.

[1920] 10. User: Confirms purchase.

[1921] 11. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[1922] 12. E-commerce: Process orders and initiate logistics processes in a short time.

[1923] 13. Server: Notifies the user of delivery status in real time.

[1924] 14. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[1925] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, reduces food waste, and improves the user experience.

[1926] The processing flow will be explained below.

[1927] Step 1:

[1928] User: Launches the application on their smartphone or PC and accesses the screen where they can input the ingredients in their refrigerator.

[1929] Step 2:

[1930] User: Enter the ingredients they currently have on hand, such as "chicken," "cabbage," and "carrots," into the text field.

[1931] Step 3:

[1932] Terminal: Convert the input ingredient list into JSON format and send it to the server.

[1933] Step 4:

[1934] Server: Analyzes the received ingredient list and converts it into a format that can be passed as input data to the generation AI.

[1935] Step 5:

[1936] Server: Pass the ingredients list to the generation AI and have it generate candidate cooking recipe information.

[1937] Step 6:

[1938] Generative AI: Based on a list of ingredients, it generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[1939] Step 7:

[1940] Server: Receives the generated recipe information and provides the video generation AI with cooking instructions and a list of necessary ingredients.

[1941] Step 8:

[1942] Video generation AI: Based on recipe information, it simulates cooking steps and generates visually easy-to-understand cooking videos.

[1943] Step 9:

[1944] Video generation AI: Returns the generated video file to the server.

[1945] Step 10:

[1946] Server: Sends the recipe information and generated video together to the user's device.

[1947] Step 11:

[1948] Terminal: Analyzes the recipe information and video sent from the server and formats them for display on the user interface.

[1949] Step 12:

[1950] Device: The user interface displays recipes such as "Stir-fried chicken" and "Cabbage and carrot salad," along with cooking videos for each recipe that can be played.

[1951] Step 13:

[1952] User: Review suggested recipes and videos and select the dish they prefer.

[1953] Step 14:

[1954] User: Sees an ingredient needed for a selected recipe that is not in the refrigerator (e.g., soy sauce) and asks to purchase it within the app.

[1955] Step 15:

[1956] Terminal: Convert the list of ingredients desired for purchase into JSON format and send it to the EC site's API.

[1957] Step 16:

[1958] Server: Obtains food inventory and price information from the EC site's API and presents it to the user.

[1959] Step 17:

[1960] User: Confirms purchase.

[1961] Step 18:

[1962] Terminal: Generates a request to send purchase confirmation information to the e-commerce site's API via the server.

[1963] Step 19:

[1964] Server: Sends the received request to the e-commerce site and confirms the order.

[1965] Step 20:

[1966] E-commerce sites: process orders and initiate logistics processes quickly.

[1967] Step 21:

[1968] Server: Receives real-time delivery status information from the e-commerce site and notifies the user.

[1969] Step 22:

[1970] User: Check delivery status and wait for additional ingredients to arrive.

[1971] Step 23:

[1972] User: When additional ingredients arrive, check their arrival on the device and proceed with cooking preparations.

[1973] Step 24:

[1974] User: Starts cooking the selected dish using the provided recipe and video.

[1975] The above are the specific processing steps of the system according to the present invention. This process allows users to efficiently use ingredients in their refrigerators and enjoy a variety of dishes.

[1976] Example 1

[1977] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1978] In modern households, it is difficult to efficiently use ingredients in the refrigerator to create new dishes. Furthermore, there is a lack of convenient ways to purchase additional ingredients, which limits the variety of dishes available. Furthermore, there is a lack of ways to clearly communicate cooking steps, which can lead to users making mistakes.

[1979] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1980] In this invention, the server includes a means for inputting ingredient information, a generation AI means for analyzing the input ingredient information and generating candidate recipes, a video generation means for generating cooking procedure videos based on the generated recipes, a user interface means for displaying the generated recipes and cooking procedure videos, an e-commerce means for purchasing additional ingredients, and a means for using prompts when generating candidate recipes. This allows users to efficiently use ingredients in their refrigerators, easily create new dishes, and easily purchase additional ingredients. Furthermore, the cooking procedures can be visually understood through the videos, reducing the risk of making mistakes in cooking procedures.

[1981] The "means for inputting ingredient information" is a means by which a user inputs information about ingredients stored in the refrigerator and transmits that information to the system.

[1982] The "generative AI means" is an artificial intelligence means for analyzing input ingredient information, generating appropriate prompt sentences, and generating cooking recipes based on these.

[1983] The "video generation means" is a means for generating a video including cooking procedures based on the generated cooking recipe.

[1984] The "user interface means" is a means for displaying the generated cooking recipe and cooking procedure video to the user.

[1985] An "electronic commerce means" is a means for purchasing additional ingredients that are needed, and serves to obtain inventory and price information and assist with the purchasing process.

[1986] "Method using prompt sentences" refers to a method of generating a cooking recipe by inputting a prompt sentence created based on ingredient information into a generation AI.

[1987] This invention is a system that proposes cooking recipes using ingredients in a user's refrigerator and generates cooking videos for those recipes. This system allows users to efficiently create new cooking menus. It also allows users to easily purchase additional ingredients as needed to increase the variety of dishes.

[1988] System Overview

[1989] 1. Ingredient input method:

[1990] The user launches the application on their smartphone or PC and manually inputs the ingredients in their refrigerator. This ingredient information is converted to JSON format by the device and sent to the server.

[1991] 2. Generation AI means:

[1992] The server analyzes the received ingredient information and passes the data to the generation AI to generate a cooking recipe. Based on ingredient information such as "chicken, cabbage, and carrots," the generation AI generates multiple recipes using the following example prompt:

[1993] Prompt Sentence Examples

[1994] Please suggest a recipe that can be made using the following ingredients:

[1995] chicken meat

[1996] cabbage

[1997] Carrots

[1998] The generation AI takes the above prompt as input and generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[1999] 3. Video generation method:

[2000] The server then provides the generated recipe information to a video generation AI, which generates a video with cooking instructions in a visually easy-to-understand format. A deep learning-based video generation model, such as PaddleGAN, is used to generate the video. The generated video file is then returned to the server.

[2001] 4. User Interface Means:

[2002] The server sends the acquired recipe information and the generated video to the user's device and displays them through a user interface. The user can view the suggested recipes and videos and select their favorite dishes.

[2003] 5. Electronic commerce instruments:

[2004] If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The delivery status is notified to the user in real time.

[2005] Specific examples

[2006] A specific example will be described in which the user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator.

[2007] 1. User: Opens the smartphone app and manually enters the ingredients in the refrigerator: "chicken," "cabbage," and "carrots."

[2008] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[2009] 3. Server: Analyzes the received ingredient list, inputs "chicken, cabbage, carrots" into the generation AI, and generates the following recipe.

[2010] Stir-fried chicken

[2011] Cabbage and carrot salad

[2012] 4. Server: Requests the video generation AI to generate a cooking video for the above recipe, and returns the generated video file to the server.

[2013] 5. Server: Sends the generated recipes and videos to the user's device.

[2014] 6. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[2015] 7. User: Review suggested recipes and videos and select "Chicken Stir-fry."

[2016] 8. E-commerce method: If the user does not have the necessary additional ingredients (e.g., soy sauce), the user requests to purchase the additional ingredients through the app. The device sends the list of ingredients to the e-commerce site.

[2017] 9. Server: Retrieves inventory and price information from the e-commerce site's API and presents it to the user.

[2018] 10. User: Confirms purchase.

[2019] 11. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[2020] 12. Server: Notifies the user of delivery status in real time.

[2021] 13. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[2022] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, reduces food waste, and improves the user experience.

[2023] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2024] Program processing flow

[2025] Step 1: Enter ingredient information

[2026] User: Launches the application on their smartphone or PC and manually inputs the ingredients they have in their refrigerator. They add ingredients such as "chicken," "cabbage," and "carrots" to the input fields and clicks the submit button.

[2027] Input: Information about ingredients in the refrigerator (e.g., "chicken," "cabbage," "carrots")

[2028] Terminal: Convert the input ingredient information into JSON format.

[2029] Output: JSON formatted ingredient information

[2030] json

[2031] {

[2032] "ingredients": ["chicken", "cabbage", "carrot"]

[2033] }

[2034] What happens: The device collects user-entered information and executes code to convert it into a JSON object, for example, using JavaScript or Python program code.

[2035] Step 2: Submit ingredient information

[2036] Terminal: Sends the converted JSON formatted ingredient information to the server.

[2037] Input: JSON formatted ingredient information

[2038] Output: Ingredient information sent to the server

[2039] What it does: Executes code to create an HTTP POST request and send it to the specified URL on the server.

[2040] Step 3: Recipe generation

[2041] Server: Analyze the received JSON formatted ingredient information.

[2042] Input: JSON formatted ingredient information

[2043] Output: A list of ingredients

[2044] What it does: Uses a JSON parsing library to get the list of ingredients.

[2045] Server: Enters prompt text into the generation AI based on the analyzed data.

[2046] Input: List of ingredients

[2047] Generative AI: Generates cooking recipes based on prompts.

[2048] Prompt Sentence Examples

[2049] Please suggest a recipe that can be made using the following ingredients:

[2050] chicken meat

[2051] cabbage

[2052] Carrots

[2053] Output: Multiple cooking recipes (e.g. "Stir-fried chicken" and "Cabbage and carrot salad")

[2054] Specific operation: Send a prompt to the generation AI (e.g., GPT model) and obtain the generated recipe list.

[2055] Step 4: Save the recipe information

[2056] Server: Stores the generated recipe information in a database.

[2057] Input: Multiple cooking recipes

[2058] Output: Recipe information stored in the database

[2059] What it does: Executes code to add recipe information to the database using an SQL INSERT statement.

[2060] Step 5: Generate cooking videos

[2061] Server: Provides recipe information to the video generation AI.

[2062] Input: Cooking recipe

[2063] Output: Cooking videos generated by video generation AI

[2064] Specific operation: Create an HTTP POST request and send recipe data to the video generation AI API.

[2065] Video generation AI: Generates videos including cooking instructions based on the provided recipe information.

[2066] Input: Cooking recipe

[2067] Output: Cooking videos

[2068] What it does: Runs code that uses a deep learning model to generate a series of image frames and stitches them together into a video.

[2069] Step 6: Submit your recipe and video

[2070] Server: Sends the generated recipe information and cooking videos to the user's device.

[2071] Input: Cooking recipes, cooking videos

[2072] Output: Recipe information and video sent to the user's device

[2073] Specific behavior: Executes code to send data to the user's device via an HTTP GET or POST request.

[2074] Step 7: Display in the user interface

[2075] Terminal: Parses the received data and displays it in the application's user interface.

[2076] Input: Cooking recipes, cooking videos

[2077] Output: Recipe information and video displayed in the user interface

[2078] Specific operation: Analyzes the received JSON data, displays the recipe title, ingredients, and cooking steps in text format, and simultaneously executes code to play the cooking video on a video player.

[2079] Step 8: Purchase additional ingredients

[2080] User: If an additional ingredient (e.g. soy sauce) is needed, the user wishes to purchase the additional ingredient in the app. Click the purchase button and select the required ingredient.

[2081] Terminal: Send the list of ingredients you wish to purchase in JSON format to the e-commerce site.

[2082] Input: List of ingredients you wish to purchase

[2083] Output: Ingredient list sent to the e-commerce site

[2084] What it does: Executes code to create an HTTP POST request and send it to the e-commerce site's API.

[2085] Step 9: Get inventory and pricing information

[2086] Server: Obtains inventory and price information from the e-commerce site's API.

[2087] Input: List of ingredients you wish to purchase

[2088] Output: Stock and price information obtained from the e-commerce site

[2089] Specific behavior: Executes code to retrieve information from the e-commerce site's API via an HTTP GET request and receive it in JSON format.

[2090] Server: Presents stock and price information to the user. Sends data to the device and displays it on the user's screen.

[2091] Input: Stock and price information obtained from the e-commerce site

[2092] Output: Stock and pricing information displayed to the user.

[2093] What it does: Sends data to the device and executes code to display it in the user interface.

[2094] Step 10: Confirm purchase and process order

[2095] User: Confirms purchase by clicking the Confirm button on the screen.

[2096] Input: Purchase confirmation instructions

[2097] Output: Confirmed information

[2098] What it does: Receives input from the user interface and executes code to send purchase confirmation information to the server.

[2099] Server: Receives purchase confirmation information, sends it to the EC site's API, confirms the order, and begins the shipping process.

[2100] Input: Purchase confirmation information

[2101] Output: Order information for the e-commerce site

[2102] What it does: Executes code to create an HTTP POST request and send it to the e-commerce site's API.

[2103] Step 11: Delivery Status Notification

[2104] Server: Notify the user of the delivery status in real time, for example via push notifications or in-app messages.

[2105] Input: Delivery status information

[2106] Output: User notification

[2107] What it does: Executes code to notify the user of delivery status using a push notification server or an in-app notification system.

[2108] As described above, this system efficiently utilizes ingredients in the user's refrigerator and provides new cooking recipes and cooking videos, thereby increasing the variety of dishes available and simplifying the purchase of additional ingredients.

[2109] (Application example 1)

[2110] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2111] Conventional recipe suggestion systems offer limited options for efficiently preparing dishes using ingredients users already have, and lack content that allows users to visually understand cooking methods. This makes it time-consuming and labor-intensive for users to create new recipes, making it difficult to efficiently utilize ingredients. Purchasing additional ingredients is also time-consuming and often burdensome for users. This limits the variety of dishes available and leads to problems such as wasted ingredients.

[2112] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2113] In this invention, the server includes a means for inputting food information, a generative AI model means for analyzing the input food information and generating candidate recipes, a video generation means for generating cooking procedure videos based on the generated recipes, a user interface means for displaying the generated recipes and cooking procedure videos, and an online commerce means for purchasing additional ingredients. This allows users to efficiently use ingredients in their refrigerators and easily create new recipes. Furthermore, by making cooking procedures easier to understand visually through videos, cooking time is reduced and additional ingredients can be purchased more smoothly. This reduces food waste and improves the user experience.

[2114] The "means for inputting food information" is an interface that allows a user to input information about ingredients and other food items in the refrigerator using a terminal that the user owns and transmits this information to the system.

[2115] The "generative AI model means" is a module that uses artificial intelligence technology to generate optimal cooking methods based on input food information.

[2116] The "video generation means" is a technology for generating a visual video of cooking procedures based on the generated recipes, so that users can easily understand how to cook.

[2117] The "user interface means" is an interface that displays the generated recipe and cooking procedure videos to the user and allows the user to view and operate them.

[2118] "Online commerce means" refers to a means by which a user can make an online transaction to purchase additional ingredients that the user needs.

[2119] This system proposes recipes using ingredients that the user has on hand, and generates and distributes videos of the cooking steps, allowing users to efficiently create new recipes and easily purchase any additional ingredients they need.

[2120] System Overview

[2121] The system includes the following means:

[2122] 1. Means for entering food information:

[2123] The user inputs the ingredients in the refrigerator using a device such as a smartphone. This ingredient information is converted into JSON format by the device and sent to the server.

[2124] 2. Generative AI model means:

[2125] The server analyzes the received ingredient information and passes the data to a generative AI to generate recipes. For example, based on ingredient information such as "chicken, cabbage, and carrots," the generative AI model generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[2126] 3. Video generation method:

[2127] The server provides the generated recipe information to the video generation AI, which generates a video with cooking instructions in a visually easy-to-understand format. The generated video file is then returned to the server.

[2128] 4. User Interface Means:

[2129] The server sends the acquired recipe information and the generated video to the user's device and displays them through a user interface. The user can view the suggested recipes and videos and select their favorite dishes.

[2130] 5. Online commerce methods:

[2131] If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The delivery status is notified to the user in real time.

[2132] Program processing

[2133] The system uses the following hardware and software:

[2134] Hardware

[2135] Smartphones and tablets (user devices)

[2136] Cloud Server

[2137] software

[2138] On the server side, recipe generation AI (e.g., GPT-4) and video generation AI (e.g., DALL-E and Deep Video) are used.

[2139] On the client side, a communication library (e.g., requests) is used to send and receive data in JSON format.

[2140] A standard web front-end framework (e.g., React, Angular, etc.) is used to display the user interface.

[2141] Specific examples

[2142] As a specific example, consider the case where the user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator.

[2143] Prompt Sentence Examples

[2144] Suggest a simple and delicious recipe using the following ingredients you have in your fridge:

[2145] chicken meat

[2146] cabbage

[2147] Carrots

[2148] The server passes this prompt to the generation AI, which generates a recipe like this:

[2149] "Stir-fried chicken" and "Cabbage and carrot salad"

[2150] The server then passes this recipe information to a video generation AI, which generates a video of the cooking steps. The generated data is stored in cloud data storage and sent to the user's device. Users can view these recipes and videos through a smartphone app and begin cooking based on the recipe they select. If additional ingredients are required, they can easily purchase them through the e-commerce site.

[2151] This system allows users to make the most of the ingredients they have, reduce waste, and easily enjoy new cooking menus.

[2152] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2153] Step 1:

[2154] The user launches the app on their smartphone and inputs the ingredients they have in their refrigerator. The user enters the specific ingredients they have (e.g., chicken, cabbage, carrots) into the app. This information is converted into a data format (e.g., JSON) and sent to the server.

[2155] Input: Ingredient information entered by the user

[2156] Output: Ingredient information in JSON format

[2157] Step 2:

[2158] The server analyzes the received ingredient information and passes the data to the generative AI model. The server then generates a prompt based on the ingredient information and inputs it into the generative AI model (e.g., GPT-4). During this analysis, the input ingredient information is formatted appropriately.

[2159] Input: JSON formatted ingredient information

[2160] Output: A prompt to be passed to the generative AI model

[2161] Step 3:

[2162] The server uses a generative AI model to generate multiple recipes based on the input prompt, which analyzes and generates several candidate recipes based on the user's ingredients.

[2163] Input: prompt statement

[2164] Output: Multiple generated recipes

[2165] Step 4:

[2166] The server provides the generated recipe information to the video generation AI, which then generates cooking instructions as videos. The video generation AI then generates video files that visually explain the steps of each recipe in an easy-to-understand format.

[2167] Input: Generated recipe information

[2168] Output: Video file visualizing the cooking steps

[2169] Step 5:

[2170] The server sends the generated recipes and videos to the user's device, stores them in cloud storage, and delivers them to the user's app in real time.

[2171] Input: Generated recipes and videos

[2172] Output: User device where recipes and videos are delivered

[2173] Step 6:

[2174] Users can browse suggested recipes and videos through the app and select their preferred dish. Users can then select one of the multiple options on the app and see how to cook it.

[2175] Input: Distributed recipes and videos

[2176] Output: User selected recipe

[2177] Step 7:

[2178] If the user wishes to purchase additional ingredients, the device sends a list of the ingredients they wish to purchase to the EC site. The device then creates a list of the ingredients needed based on the selected recipe and sends the data to the EC site.

[2179] Input: A list of additional ingredients based on the recipe selected by the user

[2180] Output: Ingredient list sent to the e-commerce site

[2181] Step 8:

[2182] The server calls the API of the e-commerce site, obtains stock and price information for each ingredient, and presents it to the user. The server analyzes the API response and presents the information in a format that is easy for the user to understand.

[2183] Input: List of additional ingredients sent

[2184] Output: Retrieved inventory and price information

[2185] Step 9:

[2186] The user confirms the purchase procedure through the app and confirms the purchase. The app receives the user's input and sends it to the server.

[2187] Input: User's purchase confirmation information

[2188] Output: Purchase confirmation information

[2189] Step 10:

[2190] The server sends the purchase confirmation information to the e-commerce site's API to complete the order process. The server also sends the order information via the API to start the logistics process.

[2191] Input: User's purchase confirmation information

[2192] Output: Order information sent to the e-commerce site

[2193] Step 11:

[2194] The server notifies the user of the delivery status in real time. The server obtains the delivery status from the EC site and notifies the user.

[2195] Input: Delivery status from the e-commerce site

[2196] Output: Real-time notification of delivery status

[2197] This process allows users to effectively use the ingredients they have and smoothly purchase any additional ingredients they need. The generated cooking video also reduces the time and effort required for cooking.

[2198] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2199] The present invention is a system that suggests cooking recipes using ingredients in the user's refrigerator and generates cooking videos for those recipes. By further combining this system with an emotion engine that recognizes the user's emotions, it is possible to suggest recipes and videos that correspond to the user's emotional state. This system not only enables users to efficiently create new cooking menus, but also provides a more satisfying experience by receiving suggestions that match their emotions. Specific program processing for this system is described below.

[2200] System Overview

[2201] 1. Ingredient input method: The user launches the application on their smartphone or PC and inputs the ingredients they have in their refrigerator. This ingredient information is converted to JSON format by the device and sent to the server.

[2202] 2. Recipe generation: The server analyzes the received ingredient information and passes the data to the generation AI to generate a cooking recipe. For example, based on ingredient information such as "chicken, cabbage, and carrots," it generates recipes such as "stir-fried chicken" or "cabbage and carrot salad."

[2203] 3. Emotion Engine: When a user is using the application, sensors such as a camera and microphone are used to analyze the user's facial expressions and voice to recognize their emotional state. This emotional information is then sent to the server.

[2204] 4. Emotion-based suggestions: The server analyzes the emotion information received from the emotion engine and adjusts the recipe suggestions based on the user's current emotional state. For example, if the user is feeling stressed, it will suggest dishes with a relaxing effect.

[2205] 5. Video generation method: The server provides recipe information and emotion information to the video generation AI, which generates a video with cooking instructions that match the emotion and is easy to understand visually. The generated video file is returned to the server.

[2206] 6. User interface means: The server sends the acquired recipe information, emotion information, and generated video to the user's device and displays them through the user interface. The user can view the suggested recipes and videos and choose their favorite dishes.

[2207] 7. E-commerce method: If the user wishes to purchase additional ingredients, the terminal sends a list of the ingredients they wish to purchase to the EC site. The server obtains inventory and price information from the EC site and presents it to the user. After the purchase is confirmed, the server sends the order information to the EC site and processes the delivery. The delivery status is notified to the user in real time.

[2208] Specific examples

[2209] As a specific example, consider a case where a user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator, and the emotion engine recognizes that the user is feeling stressed.

[2210] 1. User: Opens the app on his smartphone and enters the ingredients in his refrigerator: "chicken," "cabbage," and "carrots."

[2211] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[2212] 3. Server: Analyzes the received ingredient list, inputs "chicken, cabbage, carrots" into the generation AI, and generates the following recipes ("stir-fried chicken" and "cabbage and carrot salad").

[2213] 4. Emotion engine: Analyzes the user's facial expressions and voice and notifies the server that the user is feeling stressed.

[2214] 5. Server: Analyzes information from the emotion engine and prioritizes recipes that have a relaxing effect.

[2215] 6. Video generation method: The video generation AI is requested to generate a cooking video for the "cabbage and carrot salad" recipe, which is particularly relaxing from the above recipes. The generated video file is returned to the server.

[2216] 7. Server: Sends the generated recipes and videos to the user's device.

[2217] 8. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[2218] 9. User: Review suggested relaxing recipes and videos and choose to cook.

[2219] 10. E-commerce method: If the user does not have the necessary additional ingredients (e.g., olive oil), the user requests to purchase the additional ingredients through the app. The device sends the list of ingredients to the e-commerce site.

[2220] 11. Server: Retrieves olive oil inventory and price information from the e-commerce site's API and presents it to the user.

[2221] 12. User: Confirms purchase.

[2222] 13. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[2223] 14. E-commerce: Process orders and initiate logistics processes in a short time.

[2224] 15. Server: Notifies the user of delivery status in real time.

[2225] 16. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[2226] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, and provides a more satisfying cooking experience by making suggestions based on the user's emotional state.

[2227] The processing flow will be explained below.

[2228] Step 1:

[2229] User: Launches the application on their smartphone or PC and accesses the screen where they can input the ingredients in their refrigerator.

[2230] Step 2:

[2231] User: Enter the ingredients they currently have on hand, such as "chicken," "cabbage," and "carrots," into the text field.

[2232] Step 3:

[2233] Terminal: Convert the input ingredient list into JSON format and send it to the server.

[2234] Step 4:

[2235] Server: Analyzes the received ingredient list and converts it into a format that can be passed as input data to the generation AI.

[2236] Step 5:

[2237] Server: Pass the ingredients list to the generation AI and have it generate candidate cooking recipe information.

[2238] Step 6:

[2239] Generative AI: Based on a list of ingredients, it generates multiple recipes such as "stir-fried chicken" and "cabbage and carrot salad."

[2240] Step 7:

[2241] Server: Receives the generated recipe information.

[2242] Step 8:

[2243] Emotion engine: When a user is using an application, the system captures the user's facial expressions and voice data via the camera and microphone, and analyzes this to recognize the user's emotional state.

[2244] Step 9:

[2245] Emotion engine: Sends the recognized emotional state to the server. For example, if the user is feeling stressed, it sends the data of the stress state.

[2246] Step 10:

[2247] Server: Analyzes the emotional information received from the emotion engine and adjusts recipe suggestions based on the user's current emotional state, for example, prioritizing dishes with a relaxing effect.

[2248] Step 11:

[2249] Server: Provides recipe information and emotional information to the video generation AI and asks it to generate a cooking instruction video that matches the emotion.

[2250] Step 12:

[2251] Video generation AI: Based on recipe information, it simulates cooking steps and generates visually easy-to-understand cooking videos.

[2252] Step 13:

[2253] Video generation AI: Returns the generated video file to the server.

[2254] Step 14:

[2255] Server: Sends the recipe information and generated video together to the user's device.

[2256] Step 15:

[2257] Terminal: Analyzes the recipe information and video sent from the server and formats them for display on the user interface.

[2258] Step 16:

[2259] Device: The user interface displays recipes such as "Stir-fried chicken" and "Cabbage and carrot salad," along with cooking videos for each recipe that can be played.

[2260] Step 17:

[2261] User: Review suggested recipes and videos and select the dish they prefer.

[2262] Step 18:

[2263] User: Sees an ingredient needed for a selected recipe that is not in the refrigerator (e.g., soy sauce) and asks to purchase it within the app.

[2264] Step 19:

[2265] Terminal: Convert the list of ingredients desired for purchase into JSON format and send it to the EC site's API.

[2266] Step 20:

[2267] Server: Obtains food inventory and price information from the EC site's API and presents it to the user.

[2268] Step 21:

[2269] User: Confirms purchase.

[2270] Step 22:

[2271] Terminal: Generates a request to send purchase confirmation information to the e-commerce site's API via the server.

[2272] Step 23:

[2273] Server: Sends the received request to the e-commerce site and confirms the order.

[2274] Step 24:

[2275] E-commerce sites: process orders and initiate logistics processes quickly.

[2276] Step 25:

[2277] Server: Receives real-time delivery status information from the e-commerce site and notifies the user.

[2278] Step 26:

[2279] User: Check delivery status and wait for additional ingredients to arrive.

[2280] Step 27:

[2281] User: When additional ingredients arrive, check their arrival on the device and proceed with cooking preparations.

[2282] Step 28:

[2283] User: Starts cooking the selected dish using the provided recipe and video.

[2284] The above are the specific processing steps of the system according to the present invention. This process not only allows users to efficiently use the ingredients in their refrigerator to enjoy a variety of dishes, but also provides a more satisfying cooking experience by receiving recipe suggestions that match their emotions.

[2285] Example 2

[2286] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2287] Conventional recipe suggestion systems are unable to consider the user's emotional state when proposing recipes, making it difficult to enhance the user's psychological satisfaction. Furthermore, they lack a means to provide easy-to-understand visual instructions for cooking, making them difficult to use, especially for beginners. Another issue is the complicated process of purchasing additional ingredients, which makes it difficult to start cooking smoothly.

[2288] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2289] In this invention, the server includes a means for inputting ingredient information, an artificial intelligence means for analyzing the input ingredient information and generating candidate recipes, an emotion engine means for recognizing the user's emotional state and adjusting the cooking recipe based on the recognized emotional information, a means for generating cooking procedure videos based on the generated cooking recipe, a user interface means for displaying the generated cooking recipe and cooking procedure videos, and an electronic commerce means for purchasing any additional ingredients required. This allows the user to receive recipe suggestions suited to their emotional state and to proceed with cooking while referring to the visually easy-to-understand cooking videos. It also makes it easy to purchase any additional ingredients required.

[2290] The "means for inputting ingredient information" is an interface that allows users to input ingredients in the refrigerator as text or images via a smartphone or PC application.

[2291] The "artificial intelligence means" is a system that analyzes input ingredient information and generates candidate cooking recipes using a generative AI model.

[2292] The "emotion engine means" is a function that analyzes the user's facial expressions and voice, recognizes the user's emotional state, and adjusts the cooking recipe based on the emotional information.

[2293] The "means for generating cooking procedure videos" is a technology for generating cooking procedure videos that are visually easy to understand based on the generated cooking recipe.

[2294] The "user interface means" is an interface for displaying the generated cooking recipe and cooking procedure video to the user.

[2295] "Electronic commerce means" refers to a system that provides online shopping functionality for purchasing additional ingredients needed.

[2296] The present invention is a system that suggests cooking recipes using ingredients in the user's refrigerator and generates cooking videos for those recipes. Furthermore, by combining it with an emotion engine that recognizes the user's emotional state, it suggests recipes and videos according to that emotional state. This system allows users to efficiently create new cooking menus, and by receiving suggestions that match their emotions, it increases their satisfaction. The system's program processing is explained in detail below.

[2297] System Overview

[2298] Entering ingredient information

[2299] Users enter information about ingredients in their refrigerators through a smartphone or PC application. Input methods include text entry and uploading images taken with a camera. The entered information is converted to JSON format by the device and sent to the server.

[2300] Recipe Generation

[2301] The server analyzes the received JSON-formatted ingredient information and sends a prompt to the generative AI model (e.g., GPT-3) saying, "Please suggest a cooking recipe that includes chicken, cabbage, and carrots." The generative AI model generates multiple recipes based on the received prompt.

[2302] emotion recognition

[2303] When a user is using an application, sensors such as a camera and microphone are used to record facial expressions and voice. The device then uses the recorded data to analyze the user's emotional state using an emotion recognition engine (e.g., Microsoft Azure Emotion API). The analyzed emotional information is converted into JSON format and sent to the server.

[2304] Emotion-based recipe suggestions

[2305] The server analyzes the received emotional information and adjusts the recipe suggestions based on the user's current emotional state. For example, if the user is feeling stressed, it will prioritize suggestions of cooking recipes that have a relaxing effect. As a specific example, it generates "stir-fried chicken" and "cabbage and carrot salad," and prioritizes displaying the "cabbage and carrot salad," which has the effect of relieving stress.

[2306] Cooking video generation

[2307] The server requests a video generation AI (e.g., Synthesia) to generate a video explaining the cooking steps for the specified recipe based on the emotional state and recipe information. The generated video file is sent back to the server and provided to the user.

[2308] User display and operation

[2309] The server sends the acquired recipe information, emotion information, and generated cooking videos to the user's device, which analyzes the information and displays it through a user interface. The user can then check and operate the suggested recipes and videos.

[2310] Purchase additional ingredients

[2311] If the user does not have the necessary ingredients to prepare a suggested recipe, they can request to purchase the ingredients through the app. The device sends a list of the ingredients they wish to purchase to the EC site via an e-commerce transaction method. The server retrieves inventory and price information from the EC site's API and presents it to the user. Once the user confirms the purchase, the server sends the order information to the EC site's API and confirms the order. The user is notified of the delivery status in real time.

[2312] Specific examples

[2313] As a specific example, consider a case where a user inputs that there is "chicken," "cabbage," and "carrots" in the refrigerator, and the emotion engine recognizes that the user is feeling stressed.

[2314] 1. User: Opens the app on his smartphone and enters the ingredients in his refrigerator: "chicken," "cabbage," and "carrots."

[2315] 2. Terminal: Convert the input ingredient list into JSON format and send it to the server.

[2316] 3. Server: Analyzes the received list of ingredients, inputs "chicken, cabbage, carrots" into a generation AI (e.g., GPT-3), and generates recipes such as the following ("stir-fried chicken" and "cabbage and carrot salad").

[2317] 4. Emotion engine: Analyzes the user's facial expressions and voice and notifies the server that the user is feeling stressed.

[2318] 5. Server: Analyzes information from the emotion engine and prioritizes recipes that have a relaxing effect.

[2319] 6. Video generation method: A video generation AI (e.g., Synthesia) is requested to generate a cooking video for the "cabbage and carrot salad" recipe, which is particularly relaxing from the above recipes. The generated video file is returned to the server.

[2320] 7. Server: Sends the generated recipes and videos to the user's device.

[2321] 8. Terminal: Analyzes the recipes and videos sent from the server and displays them on the user interface.

[2322] 9. User: Review suggested relaxing recipes and videos and choose to cook.

[2323] 10. E-commerce method: If you don't have the additional ingredients you need (e.g. olive oil), submit a list of ingredients you want to purchase to the e-commerce site.

[2324] 11. Server: Retrieves olive oil inventory and price information from the e-commerce site's API and presents it to the user.

[2325] 12. User: Confirms purchase.

[2326] 13. Server: Sends purchase confirmation information to the EC site's API and confirms the order.

[2327] 14. E-commerce website: Processes orders and initiates the delivery process.

[2328] 15. Server: Notifies the user of delivery status in real time.

[2329] 16. User: Once the additional ingredients arrive, the user checks their arrival on the device and starts cooking. The user follows the recipe while watching the video provided.

[2330] Prompt Sentence Examples

[2331] An example of a prompt sentence to input into the generative AI model is, "Please suggest a relaxing cooking recipe that includes chicken, cabbage, and carrots."

[2332] In this way, the present invention is a system that makes effective use of ingredients in the refrigerator, expands the variety of dishes available, and provides a more satisfying cooking experience by making suggestions based on the user's emotional state.

[2333] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2334] Ste...

Claims

1. means for inputting ingredient information; An artificial intelligence means for analyzing input ingredient information and generating candidate cooking recipes; A means for generating a video of cooking steps based on the generated cooking recipe; a user interface means for displaying the generated cooking recipe and cooking procedure video; e-commerce means to purchase any additional ingredients needed; A system including:

2. The system according to claim 1, further comprising means for supporting the purchase of additional ingredients based on the input ingredient information.

3. The system according to claim 1 , further comprising means for transmitting the generated cooking recipe and cooking procedure video to a user's terminal.

Images