system

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving purchase information; means for storing an inventory of ingredients in a database based on the purchase information; means for generating an optimal recipe based on the inventory of ingredients and a purchase record; means for transmitting the generated recipe to a user device; means for providing a voice guide and a video guide for supporting cooking by a user in real time; means for receiving a picture of a completed dish and generating a caption for posting to an SNS; and means for displaying a AI screen for posting the generated caption and picture to the SNS on the user device. AI.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's busy lifestyles, consumers want to streamline the entire process from purchasing ingredients to cooking. However, beginners and those with limited time face challenges in reducing food waste, creating balanced meals, and understanding the cooking process. Furthermore, they often lack the enjoyment of cooking and feel obligated to do so. This makes it difficult to maintain a healthy diet and increases food waste. [Means for solving the problem]

[0005] To solve these problems, the present invention provides the following means: A means for receiving information on ingredients purchased online by a user and storing it in a database; An AI system for generating optimal recipes based on the user's ingredient inventory and purchase history and sending the recipes to the user's device; A means for providing audio and video guidance to support the cooking process in real time; A means for receiving photos of the completed dish and generating captions for posting on social media using AI, and then displaying the captions and photos on the user's device as a screen for posting on social media; An AI camera for monitoring the cooking progress and an AI algorithm for improving recipes based on user feedback are also included. This allows users to have an efficient and enjoyable cooking experience, reduce food waste, and easily maintain a healthy diet.

[0006] "Purchase information" refers to detailed data such as the ingredients selected and purchased online by the user, as well as the quantity, price, and purchase date and time.

[0007] "Inventory of ingredients" is information stored in a database about the types and quantities of ingredients owned by the user.

[0008] "AI" is an artificial intelligence technology that generates recipes, assists with cooking, and assists with posting on social media based on the user's ingredient inventory, purchasing history, and feedback.

[0009] A "recipe" is a document that contains detailed instructions and ingredient information for cooking a dish using specific ingredients.

[0010] A "user terminal" is an electronic device used by a user, such as a computer, smartphone, or tablet.

[0011] "Audio and video guidance" refers to audio and video assistance that provides users with appropriate advice and instructions in real time during the cooking process.

[0012] A "caption for social media posts" is a text message that accompanies a photo of the finished dish, explaining the contents and appeal of the dish.

[0013] An "AI camera" is a camera and its AI technology that monitors cooking progress in real time and recognizes and assists the user's actions.

[0014] The "database" is an information system for storing and managing ingredient inventory information, user purchase history, and recipe generation data generated by AI.

[0015] "Feedback" refers to user-provided evaluations and opinions about the recipes used or the system as a whole. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The system of this invention aims to provide users with an efficient and enjoyable cooking experience through the following processing steps: The entire system is designed to receive information on ingredients purchased online by users, suggest optimal recipes using AI, provide real-time support during the cooking process, and allow users to enjoyably share the finished dish on social media.

[0038] Ingredient purchase and registration

[0039] When a user purchases ingredients on an online shopping site, a list of ingredients is generated. The device receives this information and sends it to the server. The server stores the received ingredient information in a database and manages ingredient inventory. For example, if a user purchases cabbage, carrots, and chicken breast, these ingredients are registered in the database.

[0040] Recipe suggestions

[0041] The server uses AI to generate optimal recipes based on the user's ingredient inventory and past purchase history. The generated recipes also take into account the user's past ratings and preferences, as well as the expiration dates of ingredients. The generated recipes are sent from the server to the user's device, and the user can select from a list of suggested recipes. For example, a "stir-fried chicken breast" using chicken breast, cabbage, and carrots may be suggested.

[0042] Cooking support

[0043] When the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking progress with an AI camera and provides appropriate audio and video guidance in real time. This allows the user to proceed step by step with specific cooking procedures, allowing even beginners to enjoy cooking with confidence. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided via audio and video.

[0044] Support for posting cooking results on social media

[0045] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to a server, where AI generates a caption for posting on social media. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media. For example, a photo of a completed "Stir-fried chicken breast" could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0046] System benefits

[0047] This system allows users to easily purchase ingredients, reduce waste, and enjoy a healthy diet. AI support makes cooking easy even for beginners, reducing stress during cooking and increasing the enjoyment of cooking. In addition, support for posting to social media allows users to easily share their cooking results, increasing their motivation. Specifically, the system is expected to enrich users' lives by supporting a series of processes, including ingredient management, recipe suggestions, cooking support, and social media posting.

[0048] The processing flow will be explained below.

[0049] Ingredient purchase and registration

[0050] Step 1:

[0051] A user purchases groceries from an online shopping site.

[0052] Step 2:

[0053] The device generates a list of purchased ingredients (e.g., cabbage, carrots, chicken breast) and sends it to the server.

[0054] Step 3:

[0055] The server stores the received ingredient information in a database and updates ingredient inventory information.

[0056] Recipe suggestions

[0057] Step 1:

[0058] The server accesses the database to obtain the user's food inventory and purchase history.

[0059] Step 2:

[0060] The server uses AI to analyze the acquired ingredients and historical data and generate the optimal recipe for the user.

[0061] Step 3:

[0062] The server sends the generated recipe to the user's terminal.

[0063] Cooking support

[0064] Step 1:

[0065] The user selects a suggested recipe.

[0066] Step 2:

[0067] The device will instruct you to start cooking and display a list of the necessary preparations.

[0068] Step 3:

[0069] The device uses an AI camera to monitor the user's cooking progress and provides real-time audio and video guidance for each cooking step.

[0070] Support for posting cooking results on social media

[0071] Step 1:

[0072] The user takes a photo of the finished dish using the device.

[0073] Step 2:

[0074] The device sends the photograph to the server.

[0075] Step 3:

[0076] The server uses AI to generate appropriate captions for the photos and send them to the user's device.

[0077] Step 4:

[0078] The device displays the generated caption and photo on a screen for posting to SNS, and the user can confirm and edit the post before posting it to SNS.

[0079] Example 1

[0080] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0081] In modern life, many people find it difficult to maintain a healthy diet in their busy daily lives. Beginners in cooking also feel anxious and stressed during the cooking process, making it difficult to smoothly navigate the entire process from selecting the right ingredients, creating recipes, and even sharing photos of the food after cooking. Furthermore, there is a need for a way to effectively use purchased ingredients without wasting them. A system is needed to solve these challenges and provide an efficient and enjoyable cooking experience.

[0082] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0083] In this invention, the server includes means for receiving purchase information, means for storing ingredient inventory in a database based on the purchase information, a generative AI model means for generating optimal recipes based on ingredient inventory and purchase history, means for sending the generated recipes to a user terminal, means for providing audio and video guidance to support the user in cooking in real time, a generative AI model means for receiving photos of the completed dish and generating captions for posting to an SNS, and means for displaying on the user terminal a preparation screen for posting the generated captions and photos to an SNS. This allows users to smoothly and efficiently perform a series of processes from ingredient management to recipe suggestions, cooking support, and sharing on an SNS.

[0084] The "means for receiving purchase information" is a device or program for acquiring information about ingredients purchased online by a user and importing it into the system.

[0085] The "means for storing ingredient inventory in a database based on purchase information" refers to a device or program for analyzing the received purchase information and registering and managing ingredient inventory information in a database.

[0086] The "generative AI model means for generating optimal recipes based on ingredient inventory and purchase history" is a module that uses artificial intelligence to automatically create recipes that meet the user's needs, using the user's ingredient inventory and past purchase history.

[0087] "Means for transmitting the generated recipe to the user terminal" refers to a device or program for transmitting the recipe generated by the AI ​​model to the terminal used by the user.

[0088] "Means for providing audio and video guidance to assist users in cooking in real time" refers to a device or program that provides audio and video guidance at appropriate times while the user is cooking, thereby assisting the cooking process.

[0089] The "generative AI model means for receiving a photo of a finished dish and generating a caption for posting on social media" is a module that utilizes artificial intelligence to receive a photo of a dish taken by a user and, based on the photo, create an appropriate caption for posting on social media.

[0090] "Means for displaying on a user's device a screen for preparing to post the generated captions and photos to a social networking site" refers to a device or program that displays on a user's device a screen that allows the user to review and edit the captions and food photos generated by the AI ​​and post them to a social networking site.

[0091] An "AI camera that monitors cooking status" is a device that combines a camera and artificial intelligence to automatically monitor the user's actions and status while cooking, and provide feedback and guidance as needed.

[0092] The "generative AI model algorithm that takes user feedback into account to improve recipes" is an algorithm that uses artificial intelligence to reflect user ratings and comments and continuously optimize the recipe suggestions.

[0093] The system of the present invention aims to provide users with an efficient and enjoyable cooking experience. The main components of the system include a server, a terminal, and an AI model.

[0094] Overall system overview

[0095] This system is designed to receive information about ingredients purchased online by users, suggest optimal recipes using a generative AI model, provide real-time support during the cooking process, and allow users to enjoy sharing the finished dish on social media. Specifically, it consists of the following steps:

[0096] Ingredient purchase and registration

[0097] When a user purchases ingredients on an online shopping site, a list of ingredients is generated. For example, if a user purchases cabbage, carrots, and chicken breasts, the information is received by the terminal and sent to the server. The server stores the received ingredient information in a database and runs a program to manage the ingredient inventory.

[0098] Recipe suggestions

[0099] The server uses a generative AI model to generate optimal recipes based on the user's ingredient inventory and past purchase history. For example, a prompt such as "Please suggest the best recipe using cabbage, carrots, and chicken breast" is input into the generative AI model. The generated recipe also takes into account the user's preferences and the expiration dates of the ingredients. The generated recipe is sent from the server to the user's device, and the user can select the suggested recipe.

[0100] Cooking support

[0101] Once the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking process with an AI camera and provides appropriate audio and video guidance in real time. This allows the user to take specific cooking steps step by step. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided via audio and video.

[0102] Support for posting cooking results on social media

[0103] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to a server, where the generative AI model generates a caption for posting on social media. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media. For example, a photo of a dish made with cabbage, carrots, and chicken breast could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0104] Hardware and software used

[0105] The main hardware used in this system includes the device operated by the user (smartphone, tablet, PC, etc.) and the AI ​​camera that monitors the cooking status, while the main software includes a database management system running on the server, a generative AI model, a recipe generation program, and audio and video guide programs.

[0106] Specific prompt examples

[0107] An example of a prompt sentence to input to the generative AI model is as follows:

[0108] "Please suggest the best recipes using the cabbage, carrots, and chicken breast that the user has purchased. Also, please guide the user through the cooking steps of the suggested recipes in real time with audio and video guides. Furthermore, please generate captions for posting on social media based on photos of the finished dishes."

[0109] This allows the system of the present invention to help users smoothly go through a series of processes, from managing ingredients to suggesting recipes, supporting cooking, and sharing on social media.

[0110] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0111] Step 1: Receive purchase information

[0112] When a user purchases ingredients from an online shopping site, a purchase completion screen is displayed. This purchase completion screen includes a list of ingredients purchased by the user. The device extracts necessary information from this screen and sends it to the server as purchase information.

[0113] Specific operation: The user completes a purchase at the online store. The device automatically extracts the ingredient list from the purchase completion screen. The device then sends the ingredient information to the server.

[0114] Input: URL and HTML content of the purchase completion screen

[0115] Output: Extracted ingredients list (e.g. cabbage, carrot, chicken breast)

[0116] Step 2: Registering ingredient data

[0117] The server analyzes the received ingredient information and registers the list of ingredients in a database. This allows ingredient inventory information to be managed for each user. For example, information on cabbage, carrots, and chicken breast is registered.

[0118] Specific operation: The server analyzes the received ingredient information and converts it into an appropriate data format. The server then registers the ingredient inventory information in the database.

[0119] Input: List of ingredients (e.g. cabbage, carrots, chicken breast)

[0120] Output: Updated database inventory information

[0121] Step 3: Recipe generation and suggestions

[0122] The server retrieves ingredient inventory information from the database and analyzes the user's past purchase history and ratings. It then uses a generative AI model to generate an optimal recipe. For example, a prompt such as "Please suggest a recipe using cabbage, carrots, and chicken breast" is input into the generative AI model. The generated recipe is then sent from the server to the user's device.

[0123] Specific operation: The server retrieves ingredient information and past purchase history from the database. It inputs prompts into the generative AI model to generate an optimal recipe. The server then sends the generated recipe to the user's device.

[0124] Input: Ingredient inventory information, past purchase history, user ratings

[0125] Output: Suggested recipe

[0126] Step 4: Cooking support

[0127] Once the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking process with an AI camera and provides real-time audio and video instructions. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided.

[0128] Specific operation: The user selects a suggested recipe on the device. The device starts audio and video guidance. The AI ​​camera monitors the user's cooking actions and provides feedback as needed.

[0129] Input: Suggested recipes, real-time camera footage

[0130] Output: Audio guide, video guide, feedback information

[0131] Step 5: Support for posting cooking results on social media

[0132] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to the server, where the generative AI model generates a caption for posting on social media. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media.

[0133] Specific operations: A user takes a photo of a dish on their device and uploads it. The device sends the photo to the server. The generative AI model generates a caption. The server sends the generated caption to the device. The user checks and edits the caption and posts it to social media.

[0134] Input: Food photo

[0135] Output: Caption for SNS posting, SNS posting preparation screen

[0136] (Application example 1)

[0137] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0138] Conventional food ingredient management and recipe suggestion systems make it difficult to use purchased ingredients efficiently and provide insufficient support during cooking, so they are unable to completely eliminate the hassle and stress that users experience when cooking. Furthermore, sharing completed dishes on social media is a self-resolved process, so support is needed to further increase user satisfaction.

[0139] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0140] In this invention, the server includes means for receiving purchase information, means for storing ingredient inventory in a database based on the purchase information, AI means for generating optimal recipes based on ingredient inventory and purchase history, means for transmitting the generated recipes to a user terminal, means for providing audio and video guidance to support the user in cooking in real time, AI means for receiving photos of the completed dish and generating captions for posting to SNS, means for displaying on the user terminal a preparation screen for posting the generated caption and photos to SNS, and AI means for collecting information in real time as the user progresses with cooking and providing individual advice. This not only allows users to use purchased ingredients efficiently and without waste, but also reduces stress by receiving support during cooking and allows them to easily share their completed dishes on SNS.

[0141] The "means for receiving purchase information" is a function for obtaining data on ingredients and products purchased by users from online shopping sites and other digital platforms and delivering that information to the processing system.

[0142] "Means for storing food ingredient inventory in a database based on purchase information" is a system for recording and managing the inventory status of ingredients and products held by the user in a digital database based on the received purchase information.

[0143] "AI method for generating optimal recipes based on ingredient inventory and purchase history" is a technology that uses artificial intelligence to generate and suggest optimal cooking recipes for users based on ingredient inventory information stored in a database and the user's past purchase history.

[0144] "Means for sending the generated recipe to the user's device" refers to a communication function for sending recipe information generated by AI to devices such as smartphones, tablets, and PCs used by users.

[0145] "Means for providing audio and video guidance to assist users in cooking in real time" refers to a system that provides audio and video instructions and guidance in real time at appropriate times as the user proceeds with cooking, thereby assisting the user in cooking.

[0146] "AI means for receiving photos of completed dishes and generating captions for posting on social media" is a function that receives photos of dishes taken by users and uses artificial intelligence to generate appropriate captions for posting on social media based on those photos.

[0147] "Means for displaying a preparation screen on the user's device for posting the generated caption and photo to SNS" is a function that combines the caption generated by AI with the photo provided by the user, and displays a preparation screen on the device that the user can check and edit before posting to SNS.

[0148] "AI means for collecting information in real time as the user cooks and providing individualized advice" refers to artificial intelligence technology that monitors the user's cooking progress in real time and provides individualized advice and instructions to the user based on that data.

[0149] The system of the present invention receives information about ingredients purchased by a user and includes a series of procedures for supporting efficient cooking based on that information. The operation of the system will be described in detail below.

[0150] First, when a user purchases ingredients from an online shopping site, the purchase information is automatically sent to the server. The server uses a means for receiving the purchase information to store the information in a database. The stored data is used in subsequent processing.

[0151] Next, AI is used to generate optimal recipes based on the ingredients' inventory and purchasing history. This AI analyzes the information stored in the database and the user's past purchasing history to suggest optimal recipes. For this purpose, a generative AI model (e.g., GPT-3 or BERT) can be used.

[0152] The generated recipes are sent from the server to the user's device, which can include a smartphone or tablet, where the user can check the suggested recipes.

[0153] Once the user selects a suggested recipe, audio and video guidance is provided to assist with cooking in real time, allowing users to receive the necessary instructions at the appropriate time while cooking. The audio and video guidance can be realized using, for example, Google Cloud Text-to-Speech or AWS Polly.

[0154] Furthermore, it also includes a means to collect information in real time as the user cooks and provide personalized advice. This involves using AI cameras such as Raspberry Pi and Google Cloud Vision to monitor the cooking process. Based on this data, specific advice is provided to the user.

[0155] Once the dish is ready, the user uploads a photo to their device. The device then sends the photo to a server, where AI generates a caption for posting on social media. The generated caption and photo are displayed on the user's device, and the user can easily post it to social media after checking and editing. Social media integration can be achieved using, for example, the Twitter API or Facebook Graph API.

[0156] As a concrete example, consider the case where a user purchases chicken breast, cabbage, and carrots online. This information is stored in a database, and the AI ​​generates a recipe based on the prompt, "Please tell me an easy and delicious recipe using chicken breast, cabbage, and carrots." This generated recipe is sent to the user's device, and the user can proceed with cooking by following the instructions.

[0157] This system allows users to use purchased ingredients efficiently and without waste. In addition, real-time support and support for posting on social media make cooking more fun and easier to share.

[0158] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0159] Step 1:

[0160] A user purchases ingredients from an online shopping site. The terminal acquires this purchase information and sends it to a server. The input is the purchase information from the online shopping site, and the output is the purchase information sent to the server. Specifically, the terminal collects a list of ingredients purchased by the user (e.g., chicken breast, cabbage, and carrots) and sends it to the server.

[0161] Step 2:

[0162] The server stores the received purchase information in a database. The input is the purchase information sent from the device, and the output is the ingredient information stored in the database. The server uses, for example, Firebase to record the purchased ingredient information in the database.

[0163] Step 3:

[0164] The server uses AI to generate optimal recipes based on ingredient information in the database and past purchase history. The input is ingredient information in the database and past purchase history, and the output is the generated recipe. Specifically, the server uses a generative AI model (e.g., GPT-3) to generate a prompt such as, "Please tell me an easy and delicious recipe using chicken breast, cabbage, and carrots," and creates the optimal recipe based on that.

[0165] Step 4:

[0166] The server sends the generated recipe to the user's terminal. The input is the generated recipe, and the output is the recipe information sent to the user's terminal. The server uses a communication protocol to send the generated recipe (e.g., "Stir-fried chicken breast with cabbage and carrots") to the user's smartphone.

[0167] Step 5:

[0168] When a user selects a recipe, the device guides the user to start cooking. The input is the user's selection, and the output is the cooking start guidance. Specifically, the device displays audio and video guides and provides the user with specific instructions, such as "Cut the chicken breast into bite-sized pieces."

[0169] Step 6:

[0170] The device uses an AI camera to monitor the cooking process in real time to provide real-time support. The input is video information during cooking, and the output is real-time advice and instructions. The device uses, for example, Raspberry Pi and Google Cloud Vision to analyze the cooking process and provide voice guidance to the user on the next steps.

[0171] Step 7:

[0172] When the food is ready, the user takes a photo of it and uploads it to the device. The input is a photo of the food, and the output is the photo data sent to the server. The user takes a photo with their smartphone and uploads it to the device.

[0173] Step 8:

[0174] The server generates a caption for posting to social media based on the received photo. The input is a photo of the dish, and the output is the generated caption. Specifically, the server uses a generative AI model to generate the caption, "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0175] Step 9:

[0176] The server sends the generated caption and photo to the user's device and displays a preparation screen for posting to SNS. The input is the generated caption and a photo of the dish, and the output is the preparation screen. The user can then check and edit the caption and post it to SNS.

[0177] Through these processing steps, users can use purchased ingredients efficiently and without waste. In addition, real-time support and support for posting on social media make cooking more enjoyable and reduce the effort required for sharing.

[0178] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0179] The system of this invention is designed to provide users with an efficient and enjoyable cooking experience. By combining the system with an emotion engine, it is possible to take into account the user's emotions and moods and provide more personalized assistance. The entire system is designed to receive information on ingredients purchased online by the user, use AI to suggest optimal recipes, provide real-time assistance with the cooking process, and allow users to enjoy sharing the finished dish on social media.

[0180] Ingredient purchase and registration

[0181] When a user purchases ingredients on an online shopping site, a list of ingredients is generated. The device receives this information and sends it to the server. The server stores the received ingredient information in a database and manages ingredient inventory. For example, if a user purchases cabbage, carrots, and chicken breast, these ingredients are registered in the database.

[0182] Recipe suggestions

[0183] The server uses AI to generate optimal recipes based on the user's ingredient inventory and past purchase history. The generated recipes also take into account the user's past ratings and preferences, as well as the expiration dates of ingredients. Furthermore, an emotion engine is used to analyze the user's current emotional state and suggest recipes that suit that state. The generated recipes are sent from the server to the user's device, and the user can select from a list of suggested recipes. For example, if the user is feeling a bit tired, a quick and easy "stir-fried chicken breast" may be suggested.

[0184] Cooking support

[0185] Once the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking progress with an AI camera and provides appropriate audio and video guidance in real time. It uses an emotion engine to recognize the user's emotions while cooking, and adjusts the content of the audio guidance, such as providing words of encouragement, if the user feels stressed. This allows the user to proceed step by step with specific cooking procedures, allowing even beginners to enjoy cooking with confidence. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided via audio and video.

[0186] Support for posting cooking results on social media

[0187] Once the dish is complete, the user takes a photo and uploads it to their device. The device then sends the photo to a server, where AI generates a caption to post on social media. The caption is generated taking into account the user's emotional state, so if the user is feeling happy, a positive message such as "It turned out delicious!" will be added. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media. For example, a photo of a completed "Stir-fried chicken breast" could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0188] System benefits

[0189] This system allows users to easily purchase ingredients, reduce waste, and enjoy a healthy diet. The introduction of an emotion engine allows the system to take the user's emotions into consideration during cooking and when suggesting recipes, providing personalized support tailored to each individual user. This reduces the stress users feel during the cooking process and makes cooking even more enjoyable. In addition, support for posting to social media allows users to easily share the results of their cooking, increasing their motivation. Specifically, the system is expected to enrich users' lives by supporting a series of processes, including ingredient management, recipe suggestions, cooking support, and social media posting.

[0190] The processing flow will be explained below.

[0191] Ingredient purchase and registration

[0192] Step 1:

[0193] A user purchases groceries from an online shopping site.

[0194] Step 2:

[0195] The device generates a list of purchased ingredients (e.g., cabbage, carrots, chicken breast) and sends it to the server.

[0196] Step 3:

[0197] The server stores the received ingredient information in a database and updates ingredient inventory information.

[0198] Recipe suggestions

[0199] Step 1:

[0200] The server accesses the database to obtain the user's food inventory and purchase history.

[0201] Step 2:

[0202] The server uses AI to analyze the acquired ingredients and historical data and generate the optimal recipe for the user.

[0203] Step 3:

[0204] The server uses an emotion engine to recognize the user's current emotional state, which is collected through the device's camera and voice input device.

[0205] Step 4:

[0206] The server suggests recipes based on the user's emotional state. For example, a tired user might be recommended a simple dish (e.g., "stir-fried chicken breast").

[0207] Step 5:

[0208] The server sends the generated recipe to the user's terminal.

[0209] Cooking support

[0210] Step 1:

[0211] The user selects a suggested recipe.

[0212] Step 2:

[0213] The device will instruct you to start cooking and display a list of the necessary preparations.

[0214] Step 3:

[0215] The device uses an AI camera to monitor the user's cooking progress.

[0216] Step 4:

[0217] The device provides real-time audio and video instructions for each cooking step, such as "Cut the chicken breast into bite-sized pieces."

[0218] Step 5:

[0219] The device uses an emotion engine to recognize the user's emotions while cooking, and if the user is feeling stressed, it will offer words of encouragement or advice.

[0220] Support for posting cooking results on social media

[0221] Step 1:

[0222] The user takes a photo of the finished dish using the device.

[0223] Step 2:

[0224] The device sends the photograph to the server.

[0225] Step 3:

[0226] The server uses AI to generate appropriate captions for photos, taking into account the user's emotional state.

[0227] Step 4:

[0228] The server sends the generated caption and photo to the user terminal.

[0229] Step 5:

[0230] The device displays the generated caption and photo on a screen for posting to SNS, and the user can confirm and edit the post before posting it to SNS.

[0231] Specific examples of programs

[0232] Example: Purchasing ingredients and suggesting recipes

[0233] A user purchases cabbage, carrots, and chicken breasts from an online shopping site.

[0234] The terminal transmits this ingredient information to the server.

[0235] The server registers cabbage, carrots, and chicken breasts in the database.

[0236] The server uses AI and an emotion engine to generate recipes based on the user's inventory information and purchase history. For example, if the user is tired, it will suggest "easy stir-fried chicken breast."

[0237] The server transmits the generated recipe to the user terminal, and the user selects the recipe.

[0238] Example: Cooking support and social media posts

[0239] The user selects "Easy Stir-fry Chicken Breast" and starts cooking.

[0240] The device displays a list of necessary preparations and monitors the cooking progress using an AI camera.

[0241] The device provides audio and video guidance such as "Cut the chicken breast into bite-sized pieces."

[0242] Along the way, the device will recognize the user's emotions and, if they are feeling stressed, will offer words of encouragement such as, "It's okay, everything is going well!"

[0243] After cooking is complete, the user takes a photo of the finished dish, which the terminal then sends to the server.

[0244] The server uses AI to generate captions such as "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!" and sends them to the user's device.

[0245] Users can review and edit photos and captions and post them to social media.

[0246] Example 2

[0247] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0248] Until now, there has been no system that comprehensively supports the entire cooking process, from purchasing ingredients, inventory management, recipe suggestions, cooking assistance, and even posting to social media. Furthermore, technology that recognizes users' emotions during cooking and personalizes assistance in real time has been insufficient. As a result, users have had to perform numerous manual operations, which often leads to stress and difficulty during cooking. The present invention aims to solve these problems and provide users with a consistent, stress-free, and enjoyable cooking experience.

[0249] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0250] In this invention, the server includes: means for receiving purchase information; means for storing ingredient inventory in a database based on the purchase information; artificial intelligence means for generating optimal recipes based on ingredient inventory and purchase history; means for transmitting the generated recipes to a user terminal; means for providing audio and video guidance to support the user in cooking in real time; means including an emotion engine that recognizes the user's emotions while cooking and adjusts the support content based on the emotions; artificial intelligence means for receiving photos of the completed dish and generating captions to post to an SNS; and means for displaying on the user terminal a preparation screen for posting the generated caption and photo to an SNS. This enables users to manage ingredients, receive optimal recipe suggestions, receive real-time cooking support, receive support based on their emotions, and smoothly post to an SNS.

[0251] The "means for receiving purchase information" is a function for obtaining information about ingredients purchased by a user on an online shopping site.

[0252] The "means for storing ingredient inventory in a database based on purchase information" is a function for recording and managing ingredient inventory information in a database using the received purchase information.

[0253] "Artificial intelligence means for generating optimal recipes based on ingredient inventory and purchase history" is an AI technology that utilizes ingredient inventory information and past purchase history stored in a database to suggest optimal recipes to users.

[0254] "Means for sending the generated recipe to the user's terminal" is a function for sending recipe information generated by the AI ​​to the user's terminal.

[0255] The "means for providing audio and video guidance to assist the user in cooking in real time" is a function for providing audio and video instructions to the user in real time while cooking.

[0256] "Means including an emotion engine that recognizes the user's emotions while cooking and adjusts the support content based on those emotions" is a function that analyzes the user's emotional state and dynamically changes and provides the support content accordingly.

[0257] "Artificial intelligence means for receiving photos of completed dishes and generating captions for posting on social media" refers to AI technology that analyzes photos of dishes uploaded by users and generates accompanying captions for posting on social media.

[0258] "Means for displaying on the user's device a preparation screen for posting the generated caption and photo to SNS" is a function for displaying on the user's device a confirmation screen for posting the caption generated by AI and the photo uploaded by the user to SNS.

[0259] An "artificial intelligence camera" is a camera device that incorporates AI technology to monitor and analyze cooking conditions in real time.

[0260] "Artificial intelligence algorithms" are computational methods for optimizing and improving functionality based on user feedback and experience.

[0261] MODE FOR CARRYING OUT THE INVENTION

[0262] This invention is a system designed to provide users with an efficient and enjoyable cooking experience. Its key components are managing ingredient purchasing information, generating and suggesting optimal recipes, providing real-time cooking support, and supporting posting to social media. The system takes into account the user's emotional state to provide more personalized support.

[0263] Ingredient purchase and registration

[0264] When a user purchases ingredients from an online shopping site, the device receives the purchase information. The device converts this information into a standard format and sends it to the server. The server stores the received ingredient information in a database and manages inventory. For example, if a user purchases cabbage, carrots, and chicken breast, this information is registered in the database and managed as inventory.

[0265] Recipe suggestions

[0266] The server uses artificial intelligence to generate optimal recipes based on the user's ingredient inventory information and past purchase history stored in a database. It also uses an emotion engine to analyze the user's current emotional state and suggests recipes accordingly. The generated recipes are sent from the server to the user's device, and the user can select one of the suggested recipes. For example, if the user is feeling a bit tired, a quick and easy "stir-fried chicken breast" may be suggested.

[0267] Cooking support

[0268] Once the user selects a suggested recipe, the device will guide them to start cooking. The device uses an AI camera to monitor the cooking progress in real time and provides audio and video guidance. It also uses an emotion engine to recognize the user's emotions while cooking and adjusts the assistance as needed. For example, if the user is feeling stressed, the device will provide encouraging words. Specifically, it will provide audio and video instructions such as "Cut the chicken breast into bite-sized pieces."

[0269] Support for posting cooking results on social media

[0270] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to a server, which uses a generative AI model to generate a caption for the social media post. The caption is generated taking into account the user's emotional state. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it to social media. For example, a photo of a completed "Stir-fried chicken breast" could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0271] Prompt Sentence Examples

[0272] The following is an example of a prompt sentence for generating a caption to post on social media. Assume that the emotional state is "joy," the dish is "stir-fried chicken breast," and the ingredients used are cabbage and carrots.

[0273] Generate a caption for a user to post on social media. The emotional state is joy. The dish is "Stir-fried chicken breast" and the ingredients are cabbage and carrots.

[0274] An example of the generated caption is:

[0275] "I finished this stir-fried chicken breast dish with lots of fresh cabbage and carrots! It was delicious!"

[0276] This system efficiently supports users through the entire process, from managing ingredients to cooking and posting on social media. In particular, the introduction of an emotion engine enables personalized support according to the user's individual state, further improving the cooking experience.

[0277] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0278] Specific explanation of system program processing

[0279] Ingredient purchase and registration

[0280] Step 1:

[0281] A user purchases groceries from an online shopping site.

[0282] Input: Information about ingredients purchased by the user

[0283] Output: Purchased ingredients list

[0284] Step 2:

[0285] The terminal receives the purchase information and converts it into a standard format, specifically the name, quantity, and purchase date of the purchased ingredients.

[0286] Input: Purchased food list

[0287] Output: Standardized grocery shopping list

[0288] Step 3:

[0289] The terminal transmits the list of purchased ingredients to the server.

[0290] Input: Standardized grocery shopping list

[0291] Output: A request to send data to the server

[0292] Step 4:

[0293] The server stores purchase information in a database and manages food inventory. For example, purchased cabbage, carrots, and chicken breasts are recorded in the database.

[0294] Input: Standardized grocery shopping list

[0295] Output: Food inventory database

[0296] Recipe suggestions

[0297] Step 1:

[0298] The server acquires the user's food inventory information and past purchase history from the database.

[0299] Input: User ID

[0300] Output: Food inventory information, purchase history

[0301] Step 2:

[0302] Based on the information retrieved by the server, artificial intelligence methods are used to generate optimal recipes. The AI ​​model also takes into account the expiration dates of ingredients and past user ratings.

[0303] Input: Ingredient inventory information, purchase history

[0304] Output: AI-generated recipe list

[0305] Step 3:

[0306] The server uses an emotion engine to obtain the user's current emotional state and suggests recipes that match it.

[0307] Input: User emotion data

[0308] Output: Personalized recipe

[0309] Step 4:

[0310] The server sends the generated recipe to the user's terminal.

[0311] Input: Personalized Recipe

[0312] Output: Recipe displayed on user's device

[0313] Cooking support

[0314] Step 1:

[0315] The user selects a suggested recipe.

[0316] Input: Recipe list

[0317] Output: Selected recipes

[0318] Step 2:

[0319] The terminal will guide the user to start cooking.

[0320] Input: Selected recipe

[0321] Output: Cooking start notification

[0322] Step 3:

[0323] The device uses an AI camera to monitor the cooking progress in real time.

[0324] Input: Real-time cooking footage

[0325] Output: Monitoring data

[0326] Step 4:

[0327] The device provides audio and video guidance, explaining specific cooking steps to the user. At the same time, it uses an emotion engine to recognize the user's emotions while cooking and provides appropriate guidance to reduce stress.

[0328] Input: Monitoring data, real-time emotion data

[0329] Output: Audio guide, video guide

[0330] Support for posting cooking results on social media

[0331] Step 1:

[0332] The user takes a photo of the finished dish and uploads it to the device.

[0333] Input: Food photo

[0334] Output: Uploaded photo data

[0335] Step 2:

[0336] The device sends the photo to a server, which uses a generative AI model to generate a caption for the social media post.

[0337] Input: Uploaded photo data

[0338] Output: Generated captions

[0339] Step 3:

[0340] The server transmits the generated captions to the user terminal.

[0341] Input: Generated caption

[0342] Output: Captions displayed on the user's device

[0343] Step 4:

[0344] Users can review and edit the caption and photo, then post it to social media.

[0345] Input: displayed caption, photo

[0346] Output: SNS post data

[0347] Prompt Sentence Examples

[0348] The following is an example of a prompt sentence for generating a caption to post on social media. Assume that the emotional state is "joy," the dish is "stir-fried chicken breast," and the ingredients used are cabbage and carrots.

[0349] Generate a caption for a user to post on social media. The emotional state is joy. The dish is "Stir-fried chicken breast" and the ingredients are cabbage and carrots.

[0350] An example of the generated caption is:

[0351] "I finished this stir-fried chicken breast dish with lots of fresh cabbage and carrots! It was delicious!"

[0352] (Application example 2)

[0353] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0354] Conventional cooking assistance systems lack personalized support based on the user's individual emotions and moods, making the cooking process monotonous. Furthermore, physical stores lack effective support methods to improve the in-store cooking experience. There is a need for a system that can resolve the stress and problems users experience while cooking in real time and provide a more enjoyable cooking experience.

[0355] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0356] In this invention, the server includes means for receiving purchase information, means for storing ingredient inventory in a database based on the purchase information, AI means for generating optimal recipes based on ingredient inventory and purchase history, means for transmitting the generated recipes to a user terminal, means for providing audio and video guidance to support the user in cooking in real time, AI means for receiving photos of the completed dish and generating captions for posting to SNS, means for displaying on the user terminal a preparation screen for posting the generated caption and photo to SNS, means for analyzing the user's emotional state using an emotion engine and providing personalized recipe suggestions and cooking support, and means for providing customers in the store with a personalized cooking experience and cooking support in a tasting corner. This makes it possible to provide personalized cooking support based on the user's emotional state and improve the cooking experience in a physical store.

[0357] The "means for receiving purchase information" is a means for receiving information about ingredients purchased by a user from an online shopping site or an automatic scanning register and inputting the information into the system.

[0358] The "means for storing food stock in a database based on purchase information" refers to a means for recording and managing the status of the user's food stock in a database based on the received purchase information.

[0359] "AI method for generating optimal recipes based on ingredient inventory and purchase history" refers to a method that uses ingredient inventory information and past purchase history to create recipes that are most suitable for the user using AI technology.

[0360] "Means for sending the generated recipe to the user's device" refers to means for sending the recipe generated by AI to the user's device such as a smartphone or tablet via communication.

[0361] "Means for providing audio and video guidance to assist users in cooking in real time" refers to means for providing necessary procedures and advice to users in the course of cooking in real time using audio and video.

[0362] "AI means for receiving a photo of a completed dish and generating a caption for posting on social media" refers to a means for receiving a photo of a completed dish and using AI to generate an appropriate caption for posting on social media.

[0363] "Means for displaying on a user's device a preparation screen for posting the generated caption and photo to an SNS" refers to means for displaying on a user's device a preparation screen for the user to check and edit the generated caption and photo and post to an SNS.

[0364] "Means for analyzing a user's emotional state using an emotion engine and providing personalized recipe suggestions and cooking assistance" refers to means for analyzing a user's current emotional state using an emotion engine and providing individually optimized recipe suggestions and cooking assistance based on that state.

[0365] "Means for providing in-store customers with a personalized cooking experience and providing cooking support in the tasting corner" refers to means for individually optimizing the cooking experience of customers in-store and providing cooking support in the tasting corner.

[0366] This invention is a system for improving customers' cooking experience in physical stores, and includes the following functions and processing flow.

[0367] Receiving purchasing information and managing food inventory

[0368] When a user purchases ingredients at the store's automatic scanning register, the device receives the purchase information and sends it to the cloud server via the network. The server stores the received purchase information in a database and manages the user's ingredient inventory. For example, if a user purchases cabbage, carrots, and chicken breast, the information about these ingredients is recorded in the database.

[0369] Recipe suggestions

[0370] The server uses AI technology to generate optimal recipes based on ingredient inventory information and past purchase history. It also utilizes an emotion engine to analyze the user's current emotional state and suggest personalized recipes based on that. For example, if the user is tired, simple and quick recipes will be suggested. These suggested recipes are then sent to the user's device, where they can be selected.

[0371] Cooking assistance

[0372] When the user selects a suggested recipe, the device will guide them to start cooking. During cooking, an AI camera monitors the cooking progress, and the device provides real-time audio and video guidance. The device also uses an emotion engine to analyze the user's emotional state, and if the user is feeling stressed, it will provide encouraging words as audio guidance. For example, encouraging messages such as "Keep up the good work, your delicious meal will be ready soon" are played.

[0373] Support for posting finished dishes on social media

[0374] Once the dish is ready, the user takes a photo of the dish using their device and uploads it to the server. The server then uses AI to generate a caption for posting on social media. This caption is generated taking into account the user's emotional state, so it contains a positive message. For example, a caption might be generated such as, "Delicious chicken breast stir-fry with fresh cabbage and carrots is ready!" The generated caption and photo are displayed on the user's device, where the user can review and edit it before posting it to social media.

[0375] Program processing

[0376] The following hardware and software are used for server and terminal processing: a communication interface for sending and receiving information over the network, an AI engine for image and emotion analysis, and a user interface for accepting user operations. Specifically, smart glasses, tablet devices, and cloud-based servers are used.

[0377] Specific examples

[0378] For example, when a user purchases "cabbage, carrots, and chicken breast" in a store and begins cooking, the following prompt sentence is used:

[0379] Recipe suggestion prompt: "Suggest recipes based on user data and emotional state."

[0380] Sentiment analysis prompt: "Please analyze the user's emotions from this cooking face image."

[0381] Social media caption generator prompt: "Generate a positive caption based on a photo of food and your current emotional state."

[0382] This system will enable users to have a cooking experience that is tailored to their individual emotional state, and is expected to make the cooking process in physical stores even more enjoyable and stress-free.

[0383] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0384] Step 1:

[0385] Receiving purchasing information and managing food inventory

[0386] When a user purchases ingredients at the store's automatic scanning register, the terminal receives the purchase information and sends it to the cloud server via the network. The input is purchase information such as cabbage, carrots, and chicken breast, and the server stores this information in a database. This data processing ensures that the user's ingredient inventory is recorded in the database in an up-to-date state.

[0387] Step 2:

[0388] Emotion analysis

[0389] Before the user starts cooking, the device's camera takes a photo of the user's face. The input is the user's facial image, which the device sends to a cloud server. The server uses an emotion engine to analyze the user's current emotional state and sends the results back to the device. This data calculation outputs an emotional state such as "tired" or "relaxed." An example prompt is: "Please analyze the user's emotions from this facial image while cooking."

[0390] Step 3:

[0391] Recipe suggestions

[0392] The server generates an optimal recipe based on inventory information, past purchase history, and the emotional state obtained in step 2. The input is the ingredient inventory, purchase history, and emotional state, and the output is a suggested optimal recipe. The generated recipe is sent to the terminal and displayed so that the user can select it. Example prompt: "Please suggest a recommended recipe based on the user data and emotional state."

[0393] Step 4:

[0394] Cooking assistance

[0395] When the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking progress with an AI camera and provides appropriate audio and video guidance in real time. It also uses an emotion engine to analyze the user's emotional state and provide encouraging words if the user is feeling stressed. The input is video and audio of the user cooking, and the output is a guide message and encouraging message.

[0396] Step 5:

[0397] Support for taking photos of the finished dishes and posting them on social media

[0398] Once the dish is ready, the user takes a photo of the dish using their device and uploads it to the server. The input is the photo of the dish, and the server uses this to generate a caption for posting on social media. The generated caption and photo are sent back to the device for the user to review and edit, and then post to social media. The output is the generated caption and photo. Example prompt: "Generate a positive caption based on the photo of the dish and your current emotional state."

[0399] In this way, a system has been constructed in which appropriate data calculations and processing are performed based on input data at each step, and the output is provided to the user.

[0400] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0401] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0402] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0403] [Second embodiment]

[0404] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0405] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0406] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0407] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0408] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0409] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0410] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0411] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0412] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0413] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0414] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0415] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0416] The system of this invention aims to provide users with an efficient and enjoyable cooking experience through the following processing steps: The entire system is designed to receive information on ingredients purchased online by users, suggest optimal recipes using AI, provide real-time support during the cooking process, and allow users to enjoyably share the finished dish on social media.

[0417] Ingredient purchase and registration

[0418] When a user purchases ingredients on an online shopping site, a list of ingredients is generated. The device receives this information and sends it to the server. The server stores the received ingredient information in a database and manages ingredient inventory. For example, if a user purchases cabbage, carrots, and chicken breast, these ingredients are registered in the database.

[0419] Recipe suggestions

[0420] The server uses AI to generate optimal recipes based on the user's ingredient inventory and past purchase history. The generated recipes also take into account the user's past ratings and preferences, as well as the expiration dates of ingredients. The generated recipes are sent from the server to the user's device, and the user can select from a list of suggested recipes. For example, a "stir-fried chicken breast" using chicken breast, cabbage, and carrots may be suggested.

[0421] Cooking support

[0422] When the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking progress with an AI camera and provides appropriate audio and video guidance in real time. This allows the user to proceed step by step with specific cooking procedures, allowing even beginners to enjoy cooking with confidence. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided via audio and video.

[0423] Support for posting cooking results on social media

[0424] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to a server, where AI generates a caption for posting on social media. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media. For example, a photo of a completed "Stir-fried chicken breast" could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0425] System benefits

[0426] This system allows users to easily purchase ingredients, reduce waste, and enjoy a healthy diet. AI support makes cooking easy even for beginners, reducing stress during cooking and increasing the enjoyment of cooking. In addition, support for posting to social media allows users to easily share their cooking results, increasing their motivation. Specifically, the system is expected to enrich users' lives by supporting a series of processes, including ingredient management, recipe suggestions, cooking support, and social media posting.

[0427] The processing flow will be explained below.

[0428] Ingredient purchase and registration

[0429] Step 1:

[0430] A user purchases groceries from an online shopping site.

[0431] Step 2:

[0432] The device generates a list of purchased ingredients (e.g., cabbage, carrots, chicken breast) and sends it to the server.

[0433] Step 3:

[0434] The server stores the received ingredient information in a database and updates ingredient inventory information.

[0435] Recipe suggestions

[0436] Step 1:

[0437] The server accesses the database to obtain the user's food inventory and purchase history.

[0438] Step 2:

[0439] The server uses AI to analyze the acquired ingredients and historical data and generate the optimal recipe for the user.

[0440] Step 3:

[0441] The server sends the generated recipe to the user's terminal.

[0442] Cooking support

[0443] Step 1:

[0444] The user selects a suggested recipe.

[0445] Step 2:

[0446] The device will instruct you to start cooking and display a list of the necessary preparations.

[0447] Step 3:

[0448] The device uses an AI camera to monitor the user's cooking progress and provides real-time audio and video guidance for each cooking step.

[0449] Support for posting cooking results on social media

[0450] Step 1:

[0451] The user takes a photo of the finished dish using the device.

[0452] Step 2:

[0453] The device sends the photograph to the server.

[0454] Step 3:

[0455] The server uses AI to generate appropriate captions for the photos and send them to the user's device.

[0456] Step 4:

[0457] The device displays the generated caption and photo on a screen for posting to SNS, and the user can confirm and edit the post before posting it to SNS.

[0458] Example 1

[0459] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0460] In modern life, many people find it difficult to maintain a healthy diet in their busy daily lives. Beginners in cooking also feel anxious and stressed during the cooking process, making it difficult to smoothly navigate the entire process from selecting the right ingredients, creating recipes, and even sharing photos of the food after cooking. Furthermore, there is a need for a way to effectively use purchased ingredients without wasting them. A system is needed to solve these challenges and provide an efficient and enjoyable cooking experience.

[0461] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0462] In this invention, the server includes means for receiving purchase information, means for storing ingredient inventory in a database based on the purchase information, a generative AI model means for generating optimal recipes based on ingredient inventory and purchase history, means for sending the generated recipes to a user terminal, means for providing audio and video guidance to support the user in cooking in real time, a generative AI model means for receiving photos of the completed dish and generating captions for posting to an SNS, and means for displaying on the user terminal a preparation screen for posting the generated captions and photos to an SNS. This allows users to smoothly and efficiently perform a series of processes from ingredient management to recipe suggestions, cooking support, and sharing on an SNS.

[0463] The "means for receiving purchase information" is a device or program for acquiring information about ingredients purchased online by a user and importing it into the system.

[0464] The "means for storing ingredient inventory in a database based on purchase information" refers to a device or program for analyzing the received purchase information and registering and managing ingredient inventory information in a database.

[0465] The "generative AI model means for generating optimal recipes based on ingredient inventory and purchase history" is a module that uses artificial intelligence to automatically create recipes that meet the user's needs, using the user's ingredient inventory and past purchase history.

[0466] "Means for transmitting the generated recipe to the user terminal" refers to a device or program for transmitting the recipe generated by the AI ​​model to the terminal used by the user.

[0467] "Means for providing audio and video guidance to assist users in cooking in real time" refers to a device or program that provides audio and video guidance at appropriate times while the user is cooking, thereby assisting the cooking process.

[0468] The "generative AI model means for receiving a photo of a finished dish and generating a caption for posting on social media" is a module that utilizes artificial intelligence to receive a photo of a dish taken by a user and, based on the photo, create an appropriate caption for posting on social media.

[0469] "Means for displaying on a user's device a screen for preparing to post the generated captions and photos to a social networking site" refers to a device or program that displays on a user's device a screen that allows the user to review and edit the captions and food photos generated by the AI ​​and post them to a social networking site.

[0470] An "AI camera that monitors cooking status" is a device that combines a camera and artificial intelligence to automatically monitor the user's actions and status while cooking, and provide feedback and guidance as needed.

[0471] The "generative AI model algorithm that takes user feedback into account to improve recipes" is an algorithm that uses artificial intelligence to reflect user ratings and comments and continuously optimize the recipe suggestions.

[0472] The system of the present invention aims to provide users with an efficient and enjoyable cooking experience. The main components of the system include a server, a terminal, and an AI model.

[0473] Overall system overview

[0474] This system is designed to receive information about ingredients purchased online by users, suggest optimal recipes using a generative AI model, provide real-time support during the cooking process, and allow users to enjoy sharing the finished dish on social media. Specifically, it consists of the following steps:

[0475] Ingredient purchase and registration

[0476] When a user purchases ingredients on an online shopping site, a list of ingredients is generated. For example, if a user purchases cabbage, carrots, and chicken breasts, the information is received by the terminal and sent to the server. The server stores the received ingredient information in a database and runs a program to manage the ingredient inventory.

[0477] Recipe suggestions

[0478] The server uses a generative AI model to generate optimal recipes based on the user's ingredient inventory and past purchase history. For example, a prompt such as "Please suggest the best recipe using cabbage, carrots, and chicken breast" is input into the generative AI model. The generated recipe also takes into account the user's preferences and the expiration dates of the ingredients. The generated recipe is sent from the server to the user's device, and the user can select the suggested recipe.

[0479] Cooking support

[0480] Once the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking process with an AI camera and provides appropriate audio and video guidance in real time. This allows the user to take specific cooking steps step by step. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided via audio and video.

[0481] Support for posting cooking results on social media

[0482] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to a server, where the generative AI model generates a caption for posting on social media. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media. For example, a photo of a dish made with cabbage, carrots, and chicken breast could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0483] Hardware and software used

[0484] The main hardware used in this system includes the device operated by the user (smartphone, tablet, PC, etc.) and the AI ​​camera that monitors the cooking status, while the main software includes a database management system running on the server, a generative AI model, a recipe generation program, and audio and video guide programs.

[0485] Specific prompt examples

[0486] An example of a prompt sentence to input to the generative AI model is as follows:

[0487] "Please suggest the best recipes using the cabbage, carrots, and chicken breast that the user has purchased. Also, please guide the user through the cooking steps of the suggested recipes in real time with audio and video guides. Furthermore, please generate captions for posting on social media based on photos of the finished dishes."

[0488] This allows the system of the present invention to help users smoothly go through a series of processes, from managing ingredients to suggesting recipes, supporting cooking, and sharing on social media.

[0489] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0490] Step 1: Receive purchase information

[0491] When a user purchases ingredients from an online shopping site, a purchase completion screen is displayed. This purchase completion screen includes a list of ingredients purchased by the user. The device extracts necessary information from this screen and sends it to the server as purchase information.

[0492] Specific operation: The user completes a purchase at the online store. The device automatically extracts the ingredient list from the purchase completion screen. The device then sends the ingredient information to the server.

[0493] Input: URL and HTML content of the purchase completion screen

[0494] Output: Extracted ingredients list (e.g. cabbage, carrot, chicken breast)

[0495] Step 2: Registering ingredient data

[0496] The server analyzes the received ingredient information and registers the list of ingredients in a database. This allows ingredient inventory information to be managed for each user. For example, information on cabbage, carrots, and chicken breast is registered.

[0497] Specific operation: The server analyzes the received ingredient information and converts it into an appropriate data format. The server then registers the ingredient inventory information in the database.

[0498] Input: List of ingredients (e.g. cabbage, carrots, chicken breast)

[0499] Output: Updated database inventory information

[0500] Step 3: Recipe generation and suggestions

[0501] The server retrieves ingredient inventory information from the database and analyzes the user's past purchase history and ratings. It then uses a generative AI model to generate an optimal recipe. For example, a prompt such as "Please suggest a recipe using cabbage, carrots, and chicken breast" is input into the generative AI model. The generated recipe is then sent from the server to the user's device.

[0502] Specific operation: The server retrieves ingredient information and past purchase history from the database. It inputs prompts into the generative AI model to generate an optimal recipe. The server then sends the generated recipe to the user's device.

[0503] Input: Ingredient inventory information, past purchase history, user ratings

[0504] Output: Suggested recipe

[0505] Step 4: Cooking support

[0506] Once the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking process with an AI camera and provides real-time audio and video instructions. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided.

[0507] Specific operation: The user selects a suggested recipe on the device. The device starts audio and video guidance. The AI ​​camera monitors the user's cooking actions and provides feedback as needed.

[0508] Input: Suggested recipes, real-time camera footage

[0509] Output: Audio guide, video guide, feedback information

[0510] Step 5: Support for posting cooking results on social media

[0511] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to the server, where the generative AI model generates a caption for posting on social media. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media.

[0512] Specific operations: A user takes a photo of a dish on their device and uploads it. The device sends the photo to the server. The generative AI model generates a caption. The server sends the generated caption to the device. The user checks and edits the caption and posts it to social media.

[0513] Input: Food photo

[0514] Output: Caption for SNS posting, SNS posting preparation screen

[0515] (Application example 1)

[0516] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0517] Conventional food ingredient management and recipe suggestion systems make it difficult to use purchased ingredients efficiently and provide insufficient support during cooking, so they are unable to completely eliminate the hassle and stress that users experience when cooking. Furthermore, sharing completed dishes on social media is a self-resolved process, so support is needed to further increase user satisfaction.

[0518] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0519] In this invention, the server includes means for receiving purchase information, means for storing ingredient inventory in a database based on the purchase information, AI means for generating optimal recipes based on ingredient inventory and purchase history, means for transmitting the generated recipes to a user terminal, means for providing audio and video guidance to support the user in cooking in real time, AI means for receiving photos of the completed dish and generating captions for posting to SNS, means for displaying on the user terminal a preparation screen for posting the generated caption and photos to SNS, and AI means for collecting information in real time as the user progresses with cooking and providing individual advice. This not only allows users to use purchased ingredients efficiently and without waste, but also reduces stress by receiving support during cooking and allows them to easily share their completed dishes on SNS.

[0520] The "means for receiving purchase information" is a function for obtaining data on ingredients and products purchased by users from online shopping sites and other digital platforms and delivering that information to the processing system.

[0521] "Means for storing food ingredient inventory in a database based on purchase information" is a system for recording and managing the inventory status of ingredients and products held by the user in a digital database based on the received purchase information.

[0522] "AI method for generating optimal recipes based on ingredient inventory and purchase history" is a technology that uses artificial intelligence to generate and suggest optimal cooking recipes for users based on ingredient inventory information stored in a database and the user's past purchase history.

[0523] "Means for sending the generated recipe to the user's device" refers to a communication function for sending recipe information generated by AI to devices such as smartphones, tablets, and PCs used by users.

[0524] "Means for providing audio and video guidance to assist users in cooking in real time" refers to a system that provides audio and video instructions and guidance in real time at appropriate times as the user proceeds with cooking, thereby assisting the user in cooking.

[0525] "AI means for receiving photos of completed dishes and generating captions for posting on social media" is a function that receives photos of dishes taken by users and uses artificial intelligence to generate appropriate captions for posting on social media based on those photos.

[0526] "Means for displaying a preparation screen on the user's device for posting the generated caption and photo to SNS" is a function that combines the caption generated by AI with the photo provided by the user, and displays a preparation screen on the device that the user can check and edit before posting to SNS.

[0527] "AI means for collecting information in real time as the user cooks and providing individualized advice" refers to artificial intelligence technology that monitors the user's cooking progress in real time and provides individualized advice and instructions to the user based on that data.

[0528] The system of the present invention receives information about ingredients purchased by a user and includes a series of procedures for supporting efficient cooking based on that information. The operation of the system will be described in detail below.

[0529] First, when a user purchases ingredients from an online shopping site, the purchase information is automatically sent to the server. The server uses a means for receiving the purchase information to store the information in a database. The stored data is used in subsequent processing.

[0530] Next, AI is used to generate optimal recipes based on the ingredients' inventory and purchasing history. This AI analyzes the information stored in the database and the user's past purchasing history to suggest optimal recipes. For this purpose, a generative AI model (e.g., GPT-3 or BERT) can be used.

[0531] The generated recipes are sent from the server to the user's device, which can include a smartphone or tablet, where the user can check the suggested recipes.

[0532] Once the user selects a suggested recipe, audio and video guidance is provided to assist with cooking in real time, allowing users to receive the necessary instructions at the appropriate time while cooking. The audio and video guidance can be realized using, for example, Google Cloud Text-to-Speech or AWS Polly.

[0533] Furthermore, it also includes a means to collect information in real time as the user cooks and provide personalized advice. This involves using AI cameras such as Raspberry Pi and Google Cloud Vision to monitor the cooking process. Based on this data, specific advice is provided to the user.

[0534] Once the dish is ready, the user uploads a photo to their device. The device then sends the photo to a server, where AI generates a caption for posting on social media. The generated caption and photo are displayed on the user's device, and the user can easily post it to social media after checking and editing. Social media integration can be achieved using, for example, the Twitter API or Facebook Graph API.

[0535] As a concrete example, consider the case where a user purchases chicken breast, cabbage, and carrots online. This information is stored in a database, and the AI ​​generates a recipe based on the prompt, "Please tell me an easy and delicious recipe using chicken breast, cabbage, and carrots." This generated recipe is sent to the user's device, and the user can proceed with cooking by following the instructions.

[0536] This system allows users to use purchased ingredients efficiently and without waste. In addition, real-time support and support for posting on social media make cooking more fun and easier to share.

[0537] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0538] Step 1:

[0539] A user purchases ingredients from an online shopping site. The terminal acquires this purchase information and sends it to a server. The input is the purchase information from the online shopping site, and the output is the purchase information sent to the server. Specifically, the terminal collects a list of ingredients purchased by the user (e.g., chicken breast, cabbage, and carrots) and sends it to the server.

[0540] Step 2:

[0541] The server stores the received purchase information in a database. The input is the purchase information sent from the device, and the output is the ingredient information stored in the database. The server uses, for example, Firebase to record the purchased ingredient information in the database.

[0542] Step 3:

[0543] The server uses AI to generate optimal recipes based on ingredient information in the database and past purchase history. The input is ingredient information in the database and past purchase history, and the output is the generated recipe. Specifically, the server uses a generative AI model (e.g., GPT-3) to generate a prompt such as, "Please tell me an easy and delicious recipe using chicken breast, cabbage, and carrots," and creates the optimal recipe based on that.

[0544] Step 4:

[0545] The server sends the generated recipe to the user's terminal. The input is the generated recipe, and the output is the recipe information sent to the user's terminal. The server uses a communication protocol to send the generated recipe (e.g., "Stir-fried chicken breast with cabbage and carrots") to the user's smartphone.

[0546] Step 5:

[0547] When a user selects a recipe, the device guides the user to start cooking. The input is the user's selection, and the output is the cooking start guidance. Specifically, the device displays audio and video guides and provides the user with specific instructions, such as "Cut the chicken breast into bite-sized pieces."

[0548] Step 6:

[0549] The device uses an AI camera to monitor the cooking process in real time to provide real-time support. The input is video information during cooking, and the output is real-time advice and instructions. The device uses, for example, Raspberry Pi and Google Cloud Vision to analyze the cooking process and provide voice guidance to the user on the next steps.

[0550] Step 7:

[0551] When the food is ready, the user takes a photo of it and uploads it to the device. The input is a photo of the food, and the output is the photo data sent to the server. The user takes a photo with their smartphone and uploads it to the device.

[0552] Step 8:

[0553] The server generates a caption for posting to social media based on the received photo. The input is a photo of the dish, and the output is the generated caption. Specifically, the server uses a generative AI model to generate the caption, "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0554] Step 9:

[0555] The server sends the generated caption and photo to the user's device and displays a preparation screen for posting to SNS. The input is the generated caption and a photo of the dish, and the output is the preparation screen. The user can then check and edit the caption and post it to SNS.

[0556] Through these processing steps, users can use purchased ingredients efficiently and without waste. In addition, real-time support and support for posting on social media make cooking more enjoyable and reduce the effort required for sharing.

[0557] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0558] The system of this invention is designed to provide users with an efficient and enjoyable cooking experience. By combining the system with an emotion engine, it is possible to take into account the user's emotions and moods and provide more personalized assistance. The entire system is designed to receive information on ingredients purchased online by the user, use AI to suggest optimal recipes, provide real-time assistance with the cooking process, and allow users to enjoy sharing the finished dish on social media.

[0559] Ingredient purchase and registration

[0560] When a user purchases ingredients on an online shopping site, a list of ingredients is generated. The device receives this information and sends it to the server. The server stores the received ingredient information in a database and manages ingredient inventory. For example, if a user purchases cabbage, carrots, and chicken breast, these ingredients are registered in the database.

[0561] Recipe suggestions

[0562] The server uses AI to generate optimal recipes based on the user's ingredient inventory and past purchase history. The generated recipes also take into account the user's past ratings and preferences, as well as the expiration dates of ingredients. Furthermore, an emotion engine is used to analyze the user's current emotional state and suggest recipes that suit that state. The generated recipes are sent from the server to the user's device, and the user can select from a list of suggested recipes. For example, if the user is feeling a bit tired, a quick and easy "stir-fried chicken breast" may be suggested.

[0563] Cooking support

[0564] Once the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking progress with an AI camera and provides appropriate audio and video guidance in real time. It uses an emotion engine to recognize the user's emotions while cooking, and adjusts the content of the audio guidance, such as providing words of encouragement, if the user feels stressed. This allows the user to proceed step by step with specific cooking procedures, allowing even beginners to enjoy cooking with confidence. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided via audio and video.

[0565] Support for posting cooking results on social media

[0566] Once the dish is complete, the user takes a photo and uploads it to their device. The device then sends the photo to a server, where AI generates a caption to post on social media. The caption is generated taking into account the user's emotional state, so if the user is feeling happy, a positive message such as "It turned out delicious!" will be added. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media. For example, a photo of a completed "Stir-fried chicken breast" could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0567] System benefits

[0568] This system allows users to easily purchase ingredients, reduce waste, and enjoy a healthy diet. The introduction of an emotion engine allows the system to take the user's emotions into consideration during cooking and when suggesting recipes, providing personalized support tailored to each individual user. This reduces the stress users feel during the cooking process and makes cooking even more enjoyable. In addition, support for posting to social media allows users to easily share the results of their cooking, increasing their motivation. Specifically, the system is expected to enrich users' lives by supporting a series of processes, including ingredient management, recipe suggestions, cooking support, and social media posting.

[0569] The processing flow will be explained below.

[0570] Ingredient purchase and registration

[0571] Step 1:

[0572] A user purchases groceries from an online shopping site.

[0573] Step 2:

[0574] The device generates a list of purchased ingredients (e.g., cabbage, carrots, chicken breast) and sends it to the server.

[0575] Step 3:

[0576] The server stores the received ingredient information in a database and updates ingredient inventory information.

[0577] Recipe suggestions

[0578] Step 1:

[0579] The server accesses the database to obtain the user's food inventory and purchase history.

[0580] Step 2:

[0581] The server uses AI to analyze the acquired ingredients and historical data and generate the optimal recipe for the user.

[0582] Step 3:

[0583] The server uses an emotion engine to recognize the user's current emotional state, which is collected through the device's camera and voice input device.

[0584] Step 4:

[0585] The server suggests recipes based on the user's emotional state. For example, a tired user might be recommended a simple dish (e.g., "stir-fried chicken breast").

[0586] Step 5:

[0587] The server sends the generated recipe to the user's terminal.

[0588] Cooking support

[0589] Step 1:

[0590] The user selects a suggested recipe.

[0591] Step 2:

[0592] The device will instruct you to start cooking and display a list of the necessary preparations.

[0593] Step 3:

[0594] The device uses an AI camera to monitor the user's cooking progress.

[0595] Step 4:

[0596] The device provides real-time audio and video instructions for each cooking step, such as "Cut the chicken breast into bite-sized pieces."

[0597] Step 5:

[0598] The device uses an emotion engine to recognize the user's emotions while cooking, and if the user is feeling stressed, it will offer words of encouragement or advice.

[0599] Support for posting cooking results on social media

[0600] Step 1:

[0601] The user takes a photo of the finished dish using the device.

[0602] Step 2:

[0603] The device sends the photograph to the server.

[0604] Step 3:

[0605] The server uses AI to generate appropriate captions for photos, taking into account the user's emotional state.

[0606] Step 4:

[0607] The server sends the generated caption and photo to the user terminal.

[0608] Step 5:

[0609] The device displays the generated caption and photo on a screen for posting to SNS, and the user can confirm and edit the post before posting it to SNS.

[0610] Specific examples of programs

[0611] Example: Purchasing ingredients and suggesting recipes

[0612] A user purchases cabbage, carrots, and chicken breasts from an online shopping site.

[0613] The terminal transmits this ingredient information to the server.

[0614] The server registers cabbage, carrots, and chicken breasts in the database.

[0615] The server uses AI and an emotion engine to generate recipes based on the user's inventory information and purchase history. For example, if the user is tired, it will suggest "easy stir-fried chicken breast."

[0616] The server transmits the generated recipe to the user terminal, and the user selects the recipe.

[0617] Example: Cooking support and social media posts

[0618] The user selects "Easy Stir-fry Chicken Breast" and starts cooking.

[0619] The device displays a list of necessary preparations and monitors the cooking progress using an AI camera.

[0620] The device provides audio and video guidance such as "Cut the chicken breast into bite-sized pieces."

[0621] Along the way, the device will recognize the user's emotions and, if they are feeling stressed, will offer words of encouragement such as, "It's okay, everything is going well!"

[0622] After cooking is complete, the user takes a photo of the finished dish, which the terminal then sends to the server.

[0623] The server uses AI to generate captions such as "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!" and sends them to the user's device.

[0624] Users can review and edit photos and captions and post them to social media.

[0625] Example 2

[0626] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0627] Until now, there has been no system that comprehensively supports the entire cooking process, from purchasing ingredients, inventory management, recipe suggestions, cooking assistance, and even posting to social media. Furthermore, technology that recognizes users' emotions during cooking and personalizes assistance in real time has been insufficient. As a result, users have had to perform numerous manual operations, which often leads to stress and difficulty during cooking. The present invention aims to solve these problems and provide users with a consistent, stress-free, and enjoyable cooking experience.

[0628] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0629] In this invention, the server includes: means for receiving purchase information; means for storing ingredient inventory in a database based on the purchase information; artificial intelligence means for generating optimal recipes based on ingredient inventory and purchase history; means for transmitting the generated recipes to a user terminal; means for providing audio and video guidance to support the user in cooking in real time; means including an emotion engine that recognizes the user's emotions while cooking and adjusts the support content based on the emotions; artificial intelligence means for receiving photos of the completed dish and generating captions to post to an SNS; and means for displaying on the user terminal a preparation screen for posting the generated caption and photo to an SNS. This enables users to manage ingredients, receive optimal recipe suggestions, receive real-time cooking support, receive support based on their emotions, and smoothly post to an SNS.

[0630] The "means for receiving purchase information" is a function for obtaining information about ingredients purchased by a user on an online shopping site.

[0631] The "means for storing ingredient inventory in a database based on purchase information" is a function for recording and managing ingredient inventory information in a database using the received purchase information.

[0632] "Artificial intelligence means for generating optimal recipes based on ingredient inventory and purchase history" is an AI technology that utilizes ingredient inventory information and past purchase history stored in a database to suggest optimal recipes to users.

[0633] "Means for sending the generated recipe to the user's terminal" is a function for sending recipe information generated by the AI ​​to the user's terminal.

[0634] The "means for providing audio and video guidance to assist the user in cooking in real time" is a function for providing audio and video instructions to the user in real time while cooking.

[0635] "Means including an emotion engine that recognizes the user's emotions while cooking and adjusts the support content based on those emotions" is a function that analyzes the user's emotional state and dynamically changes and provides the support content accordingly.

[0636] "Artificial intelligence means for receiving photos of completed dishes and generating captions for posting on social media" refers to AI technology that analyzes photos of dishes uploaded by users and generates accompanying captions for posting on social media.

[0637] "Means for displaying on the user's device a preparation screen for posting the generated caption and photo to SNS" is a function for displaying on the user's device a confirmation screen for posting the caption generated by AI and the photo uploaded by the user to SNS.

[0638] An "artificial intelligence camera" is a camera device that incorporates AI technology to monitor and analyze cooking conditions in real time.

[0639] "Artificial intelligence algorithms" are computational methods for optimizing and improving functionality based on user feedback and experience.

[0640] MODE FOR CARRYING OUT THE INVENTION

[0641] This invention is a system designed to provide users with an efficient and enjoyable cooking experience. Its key components are managing ingredient purchasing information, generating and suggesting optimal recipes, providing real-time cooking support, and supporting posting to social media. The system takes into account the user's emotional state to provide more personalized support.

[0642] Ingredient purchase and registration

[0643] When a user purchases ingredients from an online shopping site, the device receives the purchase information. The device converts this information into a standard format and sends it to the server. The server stores the received ingredient information in a database and manages inventory. For example, if a user purchases cabbage, carrots, and chicken breast, this information is registered in the database and managed as inventory.

[0644] Recipe suggestions

[0645] The server uses artificial intelligence to generate optimal recipes based on the user's ingredient inventory information and past purchase history stored in a database. It also uses an emotion engine to analyze the user's current emotional state and suggests recipes accordingly. The generated recipes are sent from the server to the user's device, and the user can select one of the suggested recipes. For example, if the user is feeling a bit tired, a quick and easy "stir-fried chicken breast" may be suggested.

[0646] Cooking support

[0647] Once the user selects a suggested recipe, the device will guide them to start cooking. The device uses an AI camera to monitor the cooking progress in real time and provides audio and video guidance. It also uses an emotion engine to recognize the user's emotions while cooking and adjusts the assistance as needed. For example, if the user is feeling stressed, the device will provide encouraging words. Specifically, it will provide audio and video instructions such as "Cut the chicken breast into bite-sized pieces."

[0648] Support for posting cooking results on social media

[0649] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to a server, which uses a generative AI model to generate a caption for the social media post. The caption is generated taking into account the user's emotional state. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it to social media. For example, a photo of a completed "Stir-fried chicken breast" could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0650] Prompt Sentence Examples

[0651] The following is an example of a prompt sentence for generating a caption to post on social media. Assume that the emotional state is "joy," the dish is "stir-fried chicken breast," and the ingredients used are cabbage and carrots.

[0652] Generate a caption for a user to post on social media. The emotional state is joy. The dish is "Stir-fried chicken breast" and the ingredients are cabbage and carrots.

[0653] An example of the generated caption is:

[0654] "I finished this stir-fried chicken breast dish with lots of fresh cabbage and carrots! It was delicious!"

[0655] This system efficiently supports users through the entire process, from managing ingredients to cooking and posting on social media. In particular, the introduction of an emotion engine enables personalized support according to the user's individual state, further improving the cooking experience.

[0656] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0657] Specific explanation of system program processing

[0658] Ingredient purchase and registration

[0659] Step 1:

[0660] A user purchases groceries from an online shopping site.

[0661] Input: Information about ingredients purchased by the user

[0662] Output: Purchased ingredients list

[0663] Step 2:

[0664] The terminal receives the purchase information and converts it into a standard format, specifically the name, quantity, and purchase date of the purchased ingredients.

[0665] Input: Purchased food list

[0666] Output: Standardized grocery shopping list

[0667] Step 3:

[0668] The terminal transmits the list of purchased ingredients to the server.

[0669] Input: Standardized grocery shopping list

[0670] Output: A request to send data to the server

[0671] Step 4:

[0672] The server stores purchase information in a database and manages food inventory. For example, purchased cabbage, carrots, and chicken breasts are recorded in the database.

[0673] Input: Standardized grocery shopping list

[0674] Output: Food inventory database

[0675] Recipe suggestions

[0676] Step 1:

[0677] The server acquires the user's food inventory information and past purchase history from the database.

[0678] Input: User ID

[0679] Output: Food inventory information, purchase history

[0680] Step 2:

[0681] Based on the information retrieved by the server, artificial intelligence methods are used to generate optimal recipes. The AI ​​model also takes into account the expiration dates of ingredients and past user ratings.

[0682] Input: Ingredient inventory information, purchase history

[0683] Output: AI-generated recipe list

[0684] Step 3:

[0685] The server uses an emotion engine to obtain the user's current emotional state and suggests recipes that match it.

[0686] Input: User emotion data

[0687] Output: Personalized recipe

[0688] Step 4:

[0689] The server sends the generated recipe to the user's terminal.

[0690] Input: Personalized Recipe

[0691] Output: Recipe displayed on user's device

[0692] Cooking support

[0693] Step 1:

[0694] The user selects a suggested recipe.

[0695] Input: Recipe list

[0696] Output: Selected recipes

[0697] Step 2:

[0698] The terminal will guide the user to start cooking.

[0699] Input: Selected recipe

[0700] Output: Cooking start notification

[0701] Step 3:

[0702] The device uses an AI camera to monitor the cooking progress in real time.

[0703] Input: Real-time cooking footage

[0704] Output: Monitoring data

[0705] Step 4:

[0706] The device provides audio and video guidance, explaining specific cooking steps to the user. At the same time, it uses an emotion engine to recognize the user's emotions while cooking and provides appropriate guidance to reduce stress.

[0707] Input: Monitoring data, real-time emotion data

[0708] Output: Audio guide, video guide

[0709] Support for posting cooking results on social media

[0710] Step 1:

[0711] The user takes a photo of the finished dish and uploads it to the device.

[0712] Input: Food photo

[0713] Output: Uploaded photo data

[0714] Step 2:

[0715] The device sends the photo to a server, which uses a generative AI model to generate a caption for the social media post.

[0716] Input: Uploaded photo data

[0717] Output: Generated captions

[0718] Step 3:

[0719] The server transmits the generated captions to the user terminal.

[0720] Input: Generated caption

[0721] Output: Captions displayed on the user's device

[0722] Step 4:

[0723] Users can review and edit the caption and photo, then post it to social media.

[0724] Input: displayed caption, photo

[0725] Output: SNS post data

[0726] Prompt Sentence Examples

[0727] The following is an example of a prompt sentence for generating a caption to post on social media. Assume that the emotional state is "joy," the dish is "stir-fried chicken breast," and the ingredients used are cabbage and carrots.

[0728] Generate a caption for a user to post on social media. The emotional state is joy. The dish is "Stir-fried chicken breast" and the ingredients are cabbage and carrots.

[0729] An example of the generated caption is:

[0730] "I finished this stir-fried chicken breast dish with lots of fresh cabbage and carrots! It was delicious!"

[0731] (Application example 2)

[0732] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0733] Conventional cooking assistance systems lack personalized support based on the user's individual emotions and moods, making the cooking process monotonous. Furthermore, physical stores lack effective support methods to improve the in-store cooking experience. There is a need for a system that can resolve the stress and problems users experience while cooking in real time and provide a more enjoyable cooking experience.

[0734] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0735] In this invention, the server includes means for receiving purchase information, means for storing ingredient inventory in a database based on the purchase information, AI means for generating optimal recipes based on ingredient inventory and purchase history, means for transmitting the generated recipes to a user terminal, means for providing audio and video guidance to support the user in cooking in real time, AI means for receiving photos of the completed dish and generating captions for posting to SNS, means for displaying on the user terminal a preparation screen for posting the generated caption and photo to SNS, means for analyzing the user's emotional state using an emotion engine and providing personalized recipe suggestions and cooking support, and means for providing customers in the store with a personalized cooking experience and cooking support in a tasting corner. This makes it possible to provide personalized cooking support based on the user's emotional state and improve the cooking experience in a physical store.

[0736] The "means for receiving purchase information" is a means for receiving information about ingredients purchased by a user from an online shopping site or an automatic scanning register and inputting the information into the system.

[0737] The "means for storing food stock in a database based on purchase information" refers to a means for recording and managing the status of the user's food stock in a database based on the received purchase information.

[0738] "AI method for generating optimal recipes based on ingredient inventory and purchase history" refers to a method that uses ingredient inventory information and past purchase history to create recipes that are most suitable for the user using AI technology.

[0739] "Means for sending the generated recipe to the user's device" refers to means for sending the recipe generated by AI to the user's device such as a smartphone or tablet via communication.

[0740] "Means for providing audio and video guidance to assist users in cooking in real time" refers to means for providing necessary procedures and advice to users in the course of cooking in real time using audio and video.

[0741] "AI means for receiving a photo of a completed dish and generating a caption for posting on social media" refers to a means for receiving a photo of a completed dish and using AI to generate an appropriate caption for posting on social media.

[0742] "Means for displaying on a user's device a preparation screen for posting the generated caption and photo to an SNS" refers to means for displaying on a user's device a preparation screen for the user to check and edit the generated caption and photo and post to an SNS.

[0743] "Means for analyzing a user's emotional state using an emotion engine and providing personalized recipe suggestions and cooking assistance" refers to means for analyzing a user's current emotional state using an emotion engine and providing individually optimized recipe suggestions and cooking assistance based on that state.

[0744] "Means for providing in-store customers with a personalized cooking experience and providing cooking support in the tasting corner" refers to means for individually optimizing the cooking experience of customers in-store and providing cooking support in the tasting corner.

[0745] This invention is a system for improving customers' cooking experience in physical stores, and includes the following functions and processing flow.

[0746] Receiving purchasing information and managing food inventory

[0747] When a user purchases ingredients at the store's automatic scanning register, the device receives the purchase information and sends it to the cloud server via the network. The server stores the received purchase information in a database and manages the user's ingredient inventory. For example, if a user purchases cabbage, carrots, and chicken breast, the information about these ingredients is recorded in the database.

[0748] Recipe suggestions

[0749] The server uses AI technology to generate optimal recipes based on ingredient inventory information and past purchase history. It also utilizes an emotion engine to analyze the user's current emotional state and suggest personalized recipes based on that. For example, if the user is tired, simple and quick recipes will be suggested. These suggested recipes are then sent to the user's device, where they can be selected.

[0750] Cooking assistance

[0751] When the user selects a suggested recipe, the device will guide them to start cooking. During cooking, an AI camera monitors the cooking progress, and the device provides real-time audio and video guidance. The device also uses an emotion engine to analyze the user's emotional state, and if the user is feeling stressed, it will provide encouraging words as audio guidance. For example, encouraging messages such as "Keep up the good work, your delicious meal will be ready soon" are played.

[0752] Support for posting finished dishes on social media

[0753] Once the dish is ready, the user takes a photo of the dish using their device and uploads it to the server. The server then uses AI to generate a caption for posting on social media. This caption is generated taking into account the user's emotional state, so it contains a positive message. For example, a caption might be generated such as, "Delicious chicken breast stir-fry with fresh cabbage and carrots is ready!" The generated caption and photo are displayed on the user's device, where the user can review and edit it before posting it to social media.

[0754] Program processing

[0755] The following hardware and software are used for server and terminal processing: a communication interface for sending and receiving information over the network, an AI engine for image and emotion analysis, and a user interface for accepting user operations. Specifically, smart glasses, tablet devices, and cloud-based servers are used.

[0756] Specific examples

[0757] For example, when a user purchases "cabbage, carrots, and chicken breast" in a store and begins cooking, the following prompt sentence is used:

[0758] Recipe suggestion prompt: "Suggest recipes based on user data and emotional state."

[0759] Sentiment analysis prompt: "Please analyze the user's emotions from this cooking face image."

[0760] Social media caption generator prompt: "Generate a positive caption based on a photo of food and your current emotional state."

[0761] This system will enable users to have a cooking experience that is tailored to their individual emotional state, and is expected to make the cooking process in physical stores even more enjoyable and stress-free.

[0762] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0763] Step 1:

[0764] Receiving purchasing information and managing food inventory

[0765] When a user purchases ingredients at the store's automatic scanning register, the terminal receives the purchase information and sends it to the cloud server via the network. The input is purchase information such as cabbage, carrots, and chicken breast, and the server stores this information in a database. This data processing ensures that the user's ingredient inventory is recorded in the database in an up-to-date state.

[0766] Step 2:

[0767] Emotion analysis

[0768] Before the user starts cooking, the device's camera takes a photo of the user's face. The input is the user's facial image, which the device sends to a cloud server. The server uses an emotion engine to analyze the user's current emotional state and sends the results back to the device. This data calculation outputs an emotional state such as "tired" or "relaxed." An example prompt is: "Please analyze the user's emotions from this facial image while cooking."

[0769] Step 3:

[0770] Recipe suggestions

[0771] The server generates an optimal recipe based on inventory information, past purchase history, and the emotional state obtained in step 2. The input is the ingredient inventory, purchase history, and emotional state, and the output is a suggested optimal recipe. The generated recipe is sent to the terminal and displayed so that the user can select it. Example prompt: "Please suggest a recommended recipe based on the user data and emotional state."

[0772] Step 4:

[0773] Cooking assistance

[0774] When the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking progress with an AI camera and provides appropriate audio and video guidance in real time. It also uses an emotion engine to analyze the user's emotional state and provide encouraging words if the user is feeling stressed. The input is video and audio of the user cooking, and the output is a guide message and encouraging message.

[0775] Step 5:

[0776] Support for taking photos of the finished dishes and posting them on social media

[0777] Once the dish is ready, the user takes a photo of the dish using their device and uploads it to the server. The input is the photo of the dish, and the server uses this to generate a caption for posting on social media. The generated caption and photo are sent back to the device for the user to review and edit, and then post to social media. The output is the generated caption and photo. Example prompt: "Generate a positive caption based on the photo of the dish and your current emotional state."

[0778] In this way, a system has been constructed in which appropriate data calculations and processing are performed based on input data at each step, and the output is provided to the user.

[0779] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0780] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0781] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0782] [Third embodiment]

[0783] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0784] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0785] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0786] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0787] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0788] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0789] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0790] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0791] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0792] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0793] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0794] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0795] The system of this invention aims to provide users with an efficient and enjoyable cooking experience through the following processing steps: The entire system is designed to receive information on ingredients purchased online by users, suggest optimal recipes using AI, provide real-time support during the cooking process, and allow users to enjoyably share the finished dish on social media.

[0796] Ingredient purchase and registration

[0797] When a user purchases ingredients on an online shopping site, a list of ingredients is generated. The device receives this information and sends it to the server. The server stores the received ingredient information in a database and manages ingredient inventory. For example, if a user purchases cabbage, carrots, and chicken breast, these ingredients are registered in the database.

[0798] Recipe suggestions

[0799] The server uses AI to generate optimal recipes based on the user's ingredient inventory and past purchase history. The generated recipes also take into account the user's past ratings and preferences, as well as the expiration dates of ingredients. The generated recipes are sent from the server to the user's device, and the user can select from a list of suggested recipes. For example, a "stir-fried chicken breast" using chicken breast, cabbage, and carrots may be suggested.

[0800] Cooking support

[0801] When the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking progress with an AI camera and provides appropriate audio and video guidance in real time. This allows the user to proceed step by step with specific cooking procedures, allowing even beginners to enjoy cooking with confidence. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided via audio and video.

[0802] Support for posting cooking results on social media

[0803] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to a server, where AI generates a caption for posting on social media. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media. For example, a photo of a completed "Stir-fried chicken breast" could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0804] System benefits

[0805] This system allows users to easily purchase ingredients, reduce waste, and enjoy a healthy diet. AI support makes cooking easy even for beginners, reducing stress during cooking and increasing the enjoyment of cooking. In addition, support for posting to social media allows users to easily share their cooking results, increasing their motivation. Specifically, the system is expected to enrich users' lives by supporting a series of processes, including ingredient management, recipe suggestions, cooking support, and social media posting.

[0806] The processing flow will be explained below.

[0807] Ingredient purchase and registration

[0808] Step 1:

[0809] A user purchases groceries from an online shopping site.

[0810] Step 2:

[0811] The device generates a list of purchased ingredients (e.g., cabbage, carrots, chicken breast) and sends it to the server.

[0812] Step 3:

[0813] The server stores the received ingredient information in a database and updates ingredient inventory information.

[0814] Recipe suggestions

[0815] Step 1:

[0816] The server accesses the database to obtain the user's food inventory and purchase history.

[0817] Step 2:

[0818] The server uses AI to analyze the acquired ingredients and historical data and generate the optimal recipe for the user.

[0819] Step 3:

[0820] The server sends the generated recipe to the user's terminal.

[0821] Cooking support

[0822] Step 1:

[0823] The user selects a suggested recipe.

[0824] Step 2:

[0825] The device will instruct you to start cooking and display a list of the necessary preparations.

[0826] Step 3:

[0827] The device uses an AI camera to monitor the user's cooking progress and provides real-time audio and video guidance for each cooking step.

[0828] Support for posting cooking results on social media

[0829] Step 1:

[0830] The user takes a photo of the finished dish using the device.

[0831] Step 2:

[0832] The device sends the photograph to the server.

[0833] Step 3:

[0834] The server uses AI to generate appropriate captions for the photos and send them to the user's device.

[0835] Step 4:

[0836] The device displays the generated caption and photo on a screen for posting to SNS, and the user can confirm and edit the post before posting it to SNS.

[0837] Example 1

[0838] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0839] In modern life, many people find it difficult to maintain a healthy diet in their busy daily lives. Beginners in cooking also feel anxious and stressed during the cooking process, making it difficult to smoothly navigate the entire process from selecting the right ingredients, creating recipes, and even sharing photos of the food after cooking. Furthermore, there is a need for a way to effectively use purchased ingredients without wasting them. A system is needed to solve these challenges and provide an efficient and enjoyable cooking experience.

[0840] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0841] In this invention, the server includes means for receiving purchase information, means for storing ingredient inventory in a database based on the purchase information, a generative AI model means for generating optimal recipes based on ingredient inventory and purchase history, means for sending the generated recipes to a user terminal, means for providing audio and video guidance to support the user in cooking in real time, a generative AI model means for receiving photos of the completed dish and generating captions for posting to an SNS, and means for displaying on the user terminal a preparation screen for posting the generated captions and photos to an SNS. This allows users to smoothly and efficiently perform a series of processes from ingredient management to recipe suggestions, cooking support, and sharing on an SNS.

[0842] The "means for receiving purchase information" is a device or program for acquiring information about ingredients purchased online by a user and importing it into the system.

[0843] The "means for storing ingredient inventory in a database based on purchase information" refers to a device or program for analyzing the received purchase information and registering and managing ingredient inventory information in a database.

[0844] The "generative AI model means for generating optimal recipes based on ingredient inventory and purchase history" is a module that uses artificial intelligence to automatically create recipes that meet the user's needs, using the user's ingredient inventory and past purchase history.

[0845] "Means for transmitting the generated recipe to the user terminal" refers to a device or program for transmitting the recipe generated by the AI ​​model to the terminal used by the user.

[0846] "Means for providing audio and video guidance to assist users in cooking in real time" refers to a device or program that provides audio and video guidance at appropriate times while the user is cooking, thereby assisting the cooking process.

[0847] The "generative AI model means for receiving a photo of a finished dish and generating a caption for posting on social media" is a module that utilizes artificial intelligence to receive a photo of a dish taken by a user and, based on the photo, create an appropriate caption for posting on social media.

[0848] "Means for displaying on a user's device a screen for preparing to post the generated captions and photos to a social networking site" refers to a device or program that displays on a user's device a screen that allows the user to review and edit the captions and food photos generated by the AI ​​and post them to a social networking site.

[0849] An "AI camera that monitors cooking status" is a device that combines a camera and artificial intelligence to automatically monitor the user's actions and status while cooking, and provide feedback and guidance as needed.

[0850] The "generative AI model algorithm that takes user feedback into account to improve recipes" is an algorithm that uses artificial intelligence to reflect user ratings and comments and continuously optimize the recipe suggestions.

[0851] The system of the present invention aims to provide users with an efficient and enjoyable cooking experience. The main components of the system include a server, a terminal, and an AI model.

[0852] Overall system overview

[0853] This system is designed to receive information about ingredients purchased online by users, suggest optimal recipes using a generative AI model, provide real-time support during the cooking process, and allow users to enjoy sharing the finished dish on social media. Specifically, it consists of the following steps:

[0854] Ingredient purchase and registration

[0855] When a user purchases ingredients on an online shopping site, a list of ingredients is generated. For example, if a user purchases cabbage, carrots, and chicken breasts, the information is received by the terminal and sent to the server. The server stores the received ingredient information in a database and runs a program to manage the ingredient inventory.

[0856] Recipe suggestions

[0857] The server uses a generative AI model to generate optimal recipes based on the user's ingredient inventory and past purchase history. For example, a prompt such as "Please suggest the best recipe using cabbage, carrots, and chicken breast" is input into the generative AI model. The generated recipe also takes into account the user's preferences and the expiration dates of the ingredients. The generated recipe is sent from the server to the user's device, and the user can select the suggested recipe.

[0858] Cooking support

[0859] Once the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking process with an AI camera and provides appropriate audio and video guidance in real time. This allows the user to take specific cooking steps step by step. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided via audio and video.

[0860] Support for posting cooking results on social media

[0861] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to a server, where the generative AI model generates a caption for posting on social media. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media. For example, a photo of a dish made with cabbage, carrots, and chicken breast could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0862] Hardware and software used

[0863] The main hardware used in this system includes the device operated by the user (smartphone, tablet, PC, etc.) and the AI ​​camera that monitors the cooking status, while the main software includes a database management system running on the server, a generative AI model, a recipe generation program, and audio and video guide programs.

[0864] Specific prompt examples

[0865] An example of a prompt sentence to input to the generative AI model is as follows:

[0866] "Please suggest the best recipes using the cabbage, carrots, and chicken breast that the user has purchased. Also, please guide the user through the cooking steps of the suggested recipes in real time with audio and video guides. Furthermore, please generate captions for posting on social media based on photos of the finished dishes."

[0867] This allows the system of the present invention to help users smoothly go through a series of processes, from managing ingredients to suggesting recipes, supporting cooking, and sharing on social media.

[0868] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0869] Step 1: Receive purchase information

[0870] When a user purchases ingredients from an online shopping site, a purchase completion screen is displayed. This purchase completion screen includes a list of ingredients purchased by the user. The device extracts necessary information from this screen and sends it to the server as purchase information.

[0871] Specific operation: The user completes a purchase at the online store. The device automatically extracts the ingredient list from the purchase completion screen. The device then sends the ingredient information to the server.

[0872] Input: URL and HTML content of the purchase completion screen

[0873] Output: Extracted ingredients list (e.g. cabbage, carrot, chicken breast)

[0874] Step 2: Registering ingredient data

[0875] The server analyzes the received ingredient information and registers the list of ingredients in a database. This allows ingredient inventory information to be managed for each user. For example, information on cabbage, carrots, and chicken breast is registered.

[0876] Specific operation: The server analyzes the received ingredient information and converts it into an appropriate data format. The server then registers the ingredient inventory information in the database.

[0877] Input: List of ingredients (e.g. cabbage, carrots, chicken breast)

[0878] Output: Updated database inventory information

[0879] Step 3: Recipe generation and suggestions

[0880] The server retrieves ingredient inventory information from the database and analyzes the user's past purchase history and ratings. It then uses a generative AI model to generate an optimal recipe. For example, a prompt such as "Please suggest a recipe using cabbage, carrots, and chicken breast" is input into the generative AI model. The generated recipe is then sent from the server to the user's device.

[0881] Specific operation: The server retrieves ingredient information and past purchase history from the database. It inputs prompts into the generative AI model to generate an optimal recipe. The server then sends the generated recipe to the user's device.

[0882] Input: Ingredient inventory information, past purchase history, user ratings

[0883] Output: Suggested recipe

[0884] Step 4: Cooking support

[0885] Once the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking process with an AI camera and provides real-time audio and video instructions. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided.

[0886] Specific operation: The user selects a suggested recipe on the device. The device starts audio and video guidance. The AI ​​camera monitors the user's cooking actions and provides feedback as needed.

[0887] Input: Suggested recipes, real-time camera footage

[0888] Output: Audio guide, video guide, feedback information

[0889] Step 5: Support for posting cooking results on social media

[0890] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to the server, where the generative AI model generates a caption for posting on social media. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media.

[0891] Specific operations: A user takes a photo of a dish on their device and uploads it. The device sends the photo to the server. The generative AI model generates a caption. The server sends the generated caption to the device. The user checks and edits the caption and posts it to social media.

[0892] Input: Food photo

[0893] Output: Caption for SNS posting, SNS posting preparation screen

[0894] (Application example 1)

[0895] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0896] Conventional food ingredient management and recipe suggestion systems make it difficult to use purchased ingredients efficiently and provide insufficient support during cooking, so they are unable to completely eliminate the hassle and stress that users experience when cooking. Furthermore, sharing completed dishes on social media is a self-resolved process, so support is needed to further increase user satisfaction.

[0897] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0898] In this invention, the server includes means for receiving purchase information, means for storing ingredient inventory in a database based on the purchase information, AI means for generating optimal recipes based on ingredient inventory and purchase history, means for transmitting the generated recipes to a user terminal, means for providing audio and video guidance to support the user in cooking in real time, AI means for receiving photos of the completed dish and generating captions for posting to SNS, means for displaying on the user terminal a preparation screen for posting the generated caption and photos to SNS, and AI means for collecting information in real time as the user progresses with cooking and providing individual advice. This not only allows users to use purchased ingredients efficiently and without waste, but also reduces stress by receiving support during cooking and allows them to easily share their completed dishes on SNS.

[0899] The "means for receiving purchase information" is a function for obtaining data on ingredients and products purchased by users from online shopping sites and other digital platforms and delivering that information to the processing system.

[0900] "Means for storing food ingredient inventory in a database based on purchase information" is a system for recording and managing the inventory status of ingredients and products held by the user in a digital database based on the received purchase information.

[0901] "AI method for generating optimal recipes based on ingredient inventory and purchase history" is a technology that uses artificial intelligence to generate and suggest optimal cooking recipes for users based on ingredient inventory information stored in a database and the user's past purchase history.

[0902] "Means for sending the generated recipe to the user's device" refers to a communication function for sending recipe information generated by AI to devices such as smartphones, tablets, and PCs used by users.

[0903] "Means for providing audio and video guidance to assist users in cooking in real time" refers to a system that provides audio and video instructions and guidance in real time at appropriate times as the user proceeds with cooking, thereby assisting the user in cooking.

[0904] "AI means for receiving photos of completed dishes and generating captions for posting on social media" is a function that receives photos of dishes taken by users and uses artificial intelligence to generate appropriate captions for posting on social media based on those photos.

[0905] "Means for displaying a preparation screen on the user's device for posting the generated caption and photo to SNS" is a function that combines the caption generated by AI with the photo provided by the user, and displays a preparation screen on the device that the user can check and edit before posting to SNS.

[0906] "AI means for collecting information in real time as the user cooks and providing individualized advice" refers to artificial intelligence technology that monitors the user's cooking progress in real time and provides individualized advice and instructions to the user based on that data.

[0907] The system of the present invention receives information about ingredients purchased by a user and includes a series of procedures for supporting efficient cooking based on that information. The operation of the system will be described in detail below.

[0908] First, when a user purchases ingredients from an online shopping site, the purchase information is automatically sent to the server. The server uses a means for receiving the purchase information to store the information in a database. The stored data is used in subsequent processing.

[0909] Next, AI is used to generate optimal recipes based on the ingredients' inventory and purchasing history. This AI analyzes the information stored in the database and the user's past purchasing history to suggest optimal recipes. For this purpose, a generative AI model (e.g., GPT-3 or BERT) can be used.

[0910] The generated recipes are sent from the server to the user's device, which can include a smartphone or tablet, where the user can check the suggested recipes.

[0911] Once the user selects a suggested recipe, audio and video guidance is provided to assist with cooking in real time, allowing users to receive the necessary instructions at the appropriate time while cooking. The audio and video guidance can be realized using, for example, Google Cloud Text-to-Speech or AWS Polly.

[0912] Furthermore, it also includes a means to collect information in real time as the user cooks and provide personalized advice. This involves using AI cameras such as Raspberry Pi and Google Cloud Vision to monitor the cooking process. Based on this data, specific advice is provided to the user.

[0913] Once the dish is ready, the user uploads a photo to their device. The device then sends the photo to a server, where AI generates a caption for posting on social media. The generated caption and photo are displayed on the user's device, and the user can easily post it to social media after checking and editing. Social media integration can be achieved using, for example, the Twitter API or Facebook Graph API.

[0914] As a concrete example, consider the case where a user purchases chicken breast, cabbage, and carrots online. This information is stored in a database, and the AI ​​generates a recipe based on the prompt, "Please tell me an easy and delicious recipe using chicken breast, cabbage, and carrots." This generated recipe is sent to the user's device, and the user can proceed with cooking by following the instructions.

[0915] This system allows users to use purchased ingredients efficiently and without waste. In addition, real-time support and support for posting on social media make cooking more fun and easier to share.

[0916] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0917] Step 1:

[0918] A user purchases ingredients from an online shopping site. The terminal acquires this purchase information and sends it to a server. The input is the purchase information from the online shopping site, and the output is the purchase information sent to the server. Specifically, the terminal collects a list of ingredients purchased by the user (e.g., chicken breast, cabbage, and carrots) and sends it to the server.

[0919] Step 2:

[0920] The server stores the received purchase information in a database. The input is the purchase information sent from the device, and the output is the ingredient information stored in the database. The server uses, for example, Firebase to record the purchased ingredient information in the database.

[0921] Step 3:

[0922] The server uses AI to generate optimal recipes based on ingredient information in the database and past purchase history. The input is ingredient information in the database and past purchase history, and the output is the generated recipe. Specifically, the server uses a generative AI model (e.g., GPT-3) to generate a prompt such as, "Please tell me an easy and delicious recipe using chicken breast, cabbage, and carrots," and creates the optimal recipe based on that.

[0923] Step 4:

[0924] The server sends the generated recipe to the user's terminal. The input is the generated recipe, and the output is the recipe information sent to the user's terminal. The server uses a communication protocol to send the generated recipe (e.g., "Stir-fried chicken breast with cabbage and carrots") to the user's smartphone.

[0925] Step 5:

[0926] When a user selects a recipe, the device guides the user to start cooking. The input is the user's selection, and the output is the cooking start guidance. Specifically, the device displays audio and video guides and provides the user with specific instructions, such as "Cut the chicken breast into bite-sized pieces."

[0927] Step 6:

[0928] The device uses an AI camera to monitor the cooking process in real time to provide real-time support. The input is video information during cooking, and the output is real-time advice and instructions. The device uses, for example, Raspberry Pi and Google Cloud Vision to analyze the cooking process and provide voice guidance to the user on the next steps.

[0929] Step 7:

[0930] When the food is ready, the user takes a photo of it and uploads it to the device. The input is a photo of the food, and the output is the photo data sent to the server. The user takes a photo with their smartphone and uploads it to the device.

[0931] Step 8:

[0932] The server generates a caption for posting to social media based on the received photo. The input is a photo of the dish, and the output is the generated caption. Specifically, the server uses a generative AI model to generate the caption, "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0933] Step 9:

[0934] The server sends the generated caption and photo to the user's device and displays a preparation screen for posting to SNS. The input is the generated caption and a photo of the dish, and the output is the preparation screen. The user can then check and edit the caption and post it to SNS.

[0935] Through these processing steps, users can use purchased ingredients efficiently and without waste. In addition, real-time support and support for posting on social media make cooking more enjoyable and reduce the effort required for sharing.

[0936] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0937] The system of this invention is designed to provide users with an efficient and enjoyable cooking experience. By combining the system with an emotion engine, it is possible to take into account the user's emotions and moods and provide more personalized assistance. The entire system is designed to receive information on ingredients purchased online by the user, use AI to suggest optimal recipes, provide real-time assistance with the cooking process, and allow users to enjoy sharing the finished dish on social media.

[0938] Ingredient purchase and registration

[0939] When a user purchases ingredients on an online shopping site, a list of ingredients is generated. The device receives this information and sends it to the server. The server stores the received ingredient information in a database and manages ingredient inventory. For example, if a user purchases cabbage, carrots, and chicken breast, these ingredients are registered in the database.

[0940] Recipe suggestions

[0941] The server uses AI to generate optimal recipes based on the user's ingredient inventory and past purchase history. The generated recipes also take into account the user's past ratings and preferences, as well as the expiration dates of ingredients. Furthermore, an emotion engine is used to analyze the user's current emotional state and suggest recipes that suit that state. The generated recipes are sent from the server to the user's device, and the user can select from a list of suggested recipes. For example, if the user is feeling a bit tired, a quick and easy "stir-fried chicken breast" may be suggested.

[0942] Cooking support

[0943] Once the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking progress with an AI camera and provides appropriate audio and video guidance in real time. It uses an emotion engine to recognize the user's emotions while cooking, and adjusts the content of the audio guidance, such as providing words of encouragement, if the user feels stressed. This allows the user to proceed step by step with specific cooking procedures, allowing even beginners to enjoy cooking with confidence. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided via audio and video.

[0944] Support for posting cooking results on social media

[0945] Once the dish is complete, the user takes a photo and uploads it to their device. The device then sends the photo to a server, where AI generates a caption to post on social media. The caption is generated taking into account the user's emotional state, so if the user is feeling happy, a positive message such as "It turned out delicious!" will be added. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media. For example, a photo of a completed "Stir-fried chicken breast" could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[0946] System benefits

[0947] This system allows users to easily purchase ingredients, reduce waste, and enjoy a healthy diet. The introduction of an emotion engine allows the system to take the user's emotions into consideration during cooking and when suggesting recipes, providing personalized support tailored to each individual user. This reduces the stress users feel during the cooking process and makes cooking even more enjoyable. In addition, support for posting to social media allows users to easily share the results of their cooking, increasing their motivation. Specifically, the system is expected to enrich users' lives by supporting a series of processes, including ingredient management, recipe suggestions, cooking support, and social media posting.

[0948] The processing flow will be explained below.

[0949] Ingredient purchase and registration

[0950] Step 1:

[0951] A user purchases groceries from an online shopping site.

[0952] Step 2:

[0953] The device generates a list of purchased ingredients (e.g., cabbage, carrots, chicken breast) and sends it to the server.

[0954] Step 3:

[0955] The server stores the received ingredient information in a database and updates ingredient inventory information.

[0956] Recipe suggestions

[0957] Step 1:

[0958] The server accesses the database to obtain the user's food inventory and purchase history.

[0959] Step 2:

[0960] The server uses AI to analyze the acquired ingredients and historical data and generate the optimal recipe for the user.

[0961] Step 3:

[0962] The server uses an emotion engine to recognize the user's current emotional state, which is collected through the device's camera and voice input device.

[0963] Step 4:

[0964] The server suggests recipes based on the user's emotional state. For example, a tired user might be recommended a simple dish (e.g., "stir-fried chicken breast").

[0965] Step 5:

[0966] The server sends the generated recipe to the user's terminal.

[0967] Cooking support

[0968] Step 1:

[0969] The user selects a suggested recipe.

[0970] Step 2:

[0971] The device will instruct you to start cooking and display a list of the necessary preparations.

[0972] Step 3:

[0973] The device uses an AI camera to monitor the user's cooking progress.

[0974] Step 4:

[0975] The device provides real-time audio and video instructions for each cooking step, such as "Cut the chicken breast into bite-sized pieces."

[0976] Step 5:

[0977] The device uses an emotion engine to recognize the user's emotions while cooking, and if the user is feeling stressed, it will offer words of encouragement or advice.

[0978] Support for posting cooking results on social media

[0979] Step 1:

[0980] The user takes a photo of the finished dish using the device.

[0981] Step 2:

[0982] The device sends the photograph to the server.

[0983] Step 3:

[0984] The server uses AI to generate appropriate captions for photos, taking into account the user's emotional state.

[0985] Step 4:

[0986] The server sends the generated caption and photo to the user terminal.

[0987] Step 5:

[0988] The device displays the generated caption and photo on a screen for posting to SNS, and the user can confirm and edit the post before posting it to SNS.

[0989] Specific examples of programs

[0990] Example: Purchasing ingredients and suggesting recipes

[0991] A user purchases cabbage, carrots, and chicken breasts from an online shopping site.

[0992] The terminal transmits this ingredient information to the server.

[0993] The server registers cabbage, carrots, and chicken breasts in the database.

[0994] The server uses AI and an emotion engine to generate recipes based on the user's inventory information and purchase history. For example, if the user is tired, it will suggest "easy stir-fried chicken breast."

[0995] The server transmits the generated recipe to the user terminal, and the user selects the recipe.

[0996] Example: Cooking support and social media posts

[0997] The user selects "Easy Stir-fry Chicken Breast" and starts cooking.

[0998] The device displays a list of necessary preparations and monitors the cooking progress using an AI camera.

[0999] The device provides audio and video guidance such as "Cut the chicken breast into bite-sized pieces."

[1000] Along the way, the device will recognize the user's emotions and, if they are feeling stressed, will offer words of encouragement such as, "It's okay, everything is going well!"

[1001] After cooking is complete, the user takes a photo of the finished dish, which the terminal then sends to the server.

[1002] The server uses AI to generate captions such as "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!" and sends them to the user's device.

[1003] Users can review and edit photos and captions and post them to social media.

[1004] Example 2

[1005] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1006] Until now, there has been no system that comprehensively supports the entire cooking process, from purchasing ingredients, inventory management, recipe suggestions, cooking assistance, and even posting to social media. Furthermore, technology that recognizes users' emotions during cooking and personalizes assistance in real time has been insufficient. As a result, users have had to perform numerous manual operations, which often leads to stress and difficulty during cooking. The present invention aims to solve these problems and provide users with a consistent, stress-free, and enjoyable cooking experience.

[1007] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1008] In this invention, the server includes: means for receiving purchase information; means for storing ingredient inventory in a database based on the purchase information; artificial intelligence means for generating optimal recipes based on ingredient inventory and purchase history; means for transmitting the generated recipes to a user terminal; means for providing audio and video guidance to support the user in cooking in real time; means including an emotion engine that recognizes the user's emotions while cooking and adjusts the support content based on the emotions; artificial intelligence means for receiving photos of the completed dish and generating captions to post to an SNS; and means for displaying on the user terminal a preparation screen for posting the generated caption and photo to an SNS. This enables users to manage ingredients, receive optimal recipe suggestions, receive real-time cooking support, receive support based on their emotions, and smoothly post to an SNS.

[1009] The "means for receiving purchase information" is a function for obtaining information about ingredients purchased by a user on an online shopping site.

[1010] The "means for storing ingredient inventory in a database based on purchase information" is a function for recording and managing ingredient inventory information in a database using the received purchase information.

[1011] "Artificial intelligence means for generating optimal recipes based on ingredient inventory and purchase history" is an AI technology that utilizes ingredient inventory information and past purchase history stored in a database to suggest optimal recipes to users.

[1012] "Means for sending the generated recipe to the user's terminal" is a function for sending recipe information generated by the AI ​​to the user's terminal.

[1013] The "means for providing audio and video guidance to assist the user in cooking in real time" is a function for providing audio and video instructions to the user in real time while cooking.

[1014] "Means including an emotion engine that recognizes the user's emotions while cooking and adjusts the support content based on those emotions" is a function that analyzes the user's emotional state and dynamically changes and provides the support content accordingly.

[1015] "Artificial intelligence means for receiving photos of completed dishes and generating captions for posting on social media" refers to AI technology that analyzes photos of dishes uploaded by users and generates accompanying captions for posting on social media.

[1016] "Means for displaying on the user's device a preparation screen for posting the generated caption and photo to SNS" is a function for displaying on the user's device a confirmation screen for posting the caption generated by AI and the photo uploaded by the user to SNS.

[1017] An "artificial intelligence camera" is a camera device that incorporates AI technology to monitor and analyze cooking conditions in real time.

[1018] "Artificial intelligence algorithms" are computational methods for optimizing and improving functionality based on user feedback and experience.

[1019] MODE FOR CARRYING OUT THE INVENTION

[1020] This invention is a system designed to provide users with an efficient and enjoyable cooking experience. Its key components are managing ingredient purchasing information, generating and suggesting optimal recipes, providing real-time cooking support, and supporting posting to social media. The system takes into account the user's emotional state to provide more personalized support.

[1021] Ingredient purchase and registration

[1022] When a user purchases ingredients from an online shopping site, the device receives the purchase information. The device converts this information into a standard format and sends it to the server. The server stores the received ingredient information in a database and manages inventory. For example, if a user purchases cabbage, carrots, and chicken breast, this information is registered in the database and managed as inventory.

[1023] Recipe suggestions

[1024] The server uses artificial intelligence to generate optimal recipes based on the user's ingredient inventory information and past purchase history stored in a database. It also uses an emotion engine to analyze the user's current emotional state and suggests recipes accordingly. The generated recipes are sent from the server to the user's device, and the user can select one of the suggested recipes. For example, if the user is feeling a bit tired, a quick and easy "stir-fried chicken breast" may be suggested.

[1025] Cooking support

[1026] Once the user selects a suggested recipe, the device will guide them to start cooking. The device uses an AI camera to monitor the cooking progress in real time and provides audio and video guidance. It also uses an emotion engine to recognize the user's emotions while cooking and adjusts the assistance as needed. For example, if the user is feeling stressed, the device will provide encouraging words. Specifically, it will provide audio and video instructions such as "Cut the chicken breast into bite-sized pieces."

[1027] Support for posting cooking results on social media

[1028] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to a server, which uses a generative AI model to generate a caption for the social media post. The caption is generated taking into account the user's emotional state. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it to social media. For example, a photo of a completed "Stir-fried chicken breast" could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[1029] Prompt Sentence Examples

[1030] The following is an example of a prompt sentence for generating a caption to post on social media. Assume that the emotional state is "joy," the dish is "stir-fried chicken breast," and the ingredients used are cabbage and carrots.

[1031] Generate a caption for a user to post on social media. The emotional state is joy. The dish is "Stir-fried chicken breast" and the ingredients are cabbage and carrots.

[1032] An example of the generated caption is:

[1033] "I finished this stir-fried chicken breast dish with lots of fresh cabbage and carrots! It was delicious!"

[1034] This system efficiently supports users through the entire process, from managing ingredients to cooking and posting on social media. In particular, the introduction of an emotion engine enables personalized support according to the user's individual state, further improving the cooking experience.

[1035] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1036] Specific explanation of system program processing

[1037] Ingredient purchase and registration

[1038] Step 1:

[1039] A user purchases groceries from an online shopping site.

[1040] Input: Information about ingredients purchased by the user

[1041] Output: Purchased ingredients list

[1042] Step 2:

[1043] The terminal receives the purchase information and converts it into a standard format, specifically the name, quantity, and purchase date of the purchased ingredients.

[1044] Input: Purchased food list

[1045] Output: Standardized grocery shopping list

[1046] Step 3:

[1047] The terminal transmits the list of purchased ingredients to the server.

[1048] Input: Standardized grocery shopping list

[1049] Output: A request to send data to the server

[1050] Step 4:

[1051] The server stores purchase information in a database and manages food inventory. For example, purchased cabbage, carrots, and chicken breasts are recorded in the database.

[1052] Input: Standardized grocery shopping list

[1053] Output: Food inventory database

[1054] Recipe suggestions

[1055] Step 1:

[1056] The server acquires the user's food inventory information and past purchase history from the database.

[1057] Input: User ID

[1058] Output: Food inventory information, purchase history

[1059] Step 2:

[1060] Based on the information retrieved by the server, artificial intelligence methods are used to generate optimal recipes. The AI ​​model also takes into account the expiration dates of ingredients and past user ratings.

[1061] Input: Ingredient inventory information, purchase history

[1062] Output: AI-generated recipe list

[1063] Step 3:

[1064] The server uses an emotion engine to obtain the user's current emotional state and suggests recipes that match it.

[1065] Input: User emotion data

[1066] Output: Personalized recipe

[1067] Step 4:

[1068] The server sends the generated recipe to the user's terminal.

[1069] Input: Personalized Recipe

[1070] Output: Recipe displayed on user's device

[1071] Cooking support

[1072] Step 1:

[1073] The user selects a suggested recipe.

[1074] Input: Recipe list

[1075] Output: Selected recipes

[1076] Step 2:

[1077] The terminal will guide the user to start cooking.

[1078] Input: Selected recipe

[1079] Output: Cooking start notification

[1080] Step 3:

[1081] The device uses an AI camera to monitor the cooking progress in real time.

[1082] Input: Real-time cooking footage

[1083] Output: Monitoring data

[1084] Step 4:

[1085] The device provides audio and video guidance, explaining specific cooking steps to the user. At the same time, it uses an emotion engine to recognize the user's emotions while cooking and provides appropriate guidance to reduce stress.

[1086] Input: Monitoring data, real-time emotion data

[1087] Output: Audio guide, video guide

[1088] Support for posting cooking results on social media

[1089] Step 1:

[1090] The user takes a photo of the finished dish and uploads it to the device.

[1091] Input: Food photo

[1092] Output: Uploaded photo data

[1093] Step 2:

[1094] The device sends the photo to a server, which uses a generative AI model to generate a caption for the social media post.

[1095] Input: Uploaded photo data

[1096] Output: Generated captions

[1097] Step 3:

[1098] The server transmits the generated captions to the user terminal.

[1099] Input: Generated caption

[1100] Output: Captions displayed on the user's device

[1101] Step 4:

[1102] Users can review and edit the caption and photo, then post it to social media.

[1103] Input: displayed caption, photo

[1104] Output: SNS post data

[1105] Prompt Sentence Examples

[1106] The following is an example of a prompt sentence for generating a caption to post on social media. Assume that the emotional state is "joy," the dish is "stir-fried chicken breast," and the ingredients used are cabbage and carrots.

[1107] Generate a caption for a user to post on social media. The emotional state is joy. The dish is "Stir-fried chicken breast" and the ingredients are cabbage and carrots.

[1108] An example of the generated caption is:

[1109] "I finished this stir-fried chicken breast dish with lots of fresh cabbage and carrots! It was delicious!"

[1110] (Application example 2)

[1111] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1112] Conventional cooking assistance systems lack personalized support based on the user's individual emotions and moods, making the cooking process monotonous. Furthermore, physical stores lack effective support methods to improve the in-store cooking experience. There is a need for a system that can resolve the stress and problems users experience while cooking in real time and provide a more enjoyable cooking experience.

[1113] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1114] In this invention, the server includes means for receiving purchase information, means for storing ingredient inventory in a database based on the purchase information, AI means for generating optimal recipes based on ingredient inventory and purchase history, means for transmitting the generated recipes to a user terminal, means for providing audio and video guidance to support the user in cooking in real time, AI means for receiving photos of the completed dish and generating captions for posting to SNS, means for displaying on the user terminal a preparation screen for posting the generated caption and photo to SNS, means for analyzing the user's emotional state using an emotion engine and providing personalized recipe suggestions and cooking support, and means for providing customers in the store with a personalized cooking experience and cooking support in a tasting corner. This makes it possible to provide personalized cooking support based on the user's emotional state and improve the cooking experience in a physical store.

[1115] The "means for receiving purchase information" is a means for receiving information about ingredients purchased by a user from an online shopping site or an automatic scanning register and inputting the information into the system.

[1116] The "means for storing food stock in a database based on purchase information" refers to a means for recording and managing the status of the user's food stock in a database based on the received purchase information.

[1117] "AI method for generating optimal recipes based on ingredient inventory and purchase history" refers to a method that uses ingredient inventory information and past purchase history to create recipes that are most suitable for the user using AI technology.

[1118] "Means for sending the generated recipe to the user's device" refers to means for sending the recipe generated by AI to the user's device such as a smartphone or tablet via communication.

[1119] "Means for providing audio and video guidance to assist users in cooking in real time" refers to means for providing necessary procedures and advice to users in the course of cooking in real time using audio and video.

[1120] "AI means for receiving a photo of a completed dish and generating a caption for posting on social media" refers to a means for receiving a photo of a completed dish and using AI to generate an appropriate caption for posting on social media.

[1121] "Means for displaying on a user's device a preparation screen for posting the generated caption and photo to an SNS" refers to means for displaying on a user's device a preparation screen for the user to check and edit the generated caption and photo and post to an SNS.

[1122] "Means for analyzing a user's emotional state using an emotion engine and providing personalized recipe suggestions and cooking assistance" refers to means for analyzing a user's current emotional state using an emotion engine and providing individually optimized recipe suggestions and cooking assistance based on that state.

[1123] "Means for providing in-store customers with a personalized cooking experience and providing cooking support in the tasting corner" refers to means for individually optimizing the cooking experience of customers in-store and providing cooking support in the tasting corner.

[1124] This invention is a system for improving customers' cooking experience in physical stores, and includes the following functions and processing flow.

[1125] Receiving purchasing information and managing food inventory

[1126] When a user purchases ingredients at the store's automatic scanning register, the device receives the purchase information and sends it to the cloud server via the network. The server stores the received purchase information in a database and manages the user's ingredient inventory. For example, if a user purchases cabbage, carrots, and chicken breast, the information about these ingredients is recorded in the database.

[1127] Recipe suggestions

[1128] The server uses AI technology to generate optimal recipes based on ingredient inventory information and past purchase history. It also utilizes an emotion engine to analyze the user's current emotional state and suggest personalized recipes based on that. For example, if the user is tired, simple and quick recipes will be suggested. These suggested recipes are then sent to the user's device, where they can be selected.

[1129] Cooking assistance

[1130] When the user selects a suggested recipe, the device will guide them to start cooking. During cooking, an AI camera monitors the cooking progress, and the device provides real-time audio and video guidance. The device also uses an emotion engine to analyze the user's emotional state, and if the user is feeling stressed, it will provide encouraging words as audio guidance. For example, encouraging messages such as "Keep up the good work, your delicious meal will be ready soon" are played.

[1131] Support for posting finished dishes on social media

[1132] Once the dish is ready, the user takes a photo of the dish using their device and uploads it to the server. The server then uses AI to generate a caption for posting on social media. This caption is generated taking into account the user's emotional state, so it contains a positive message. For example, a caption might be generated such as, "Delicious chicken breast stir-fry with fresh cabbage and carrots is ready!" The generated caption and photo are displayed on the user's device, where the user can review and edit it before posting it to social media.

[1133] Program processing

[1134] The following hardware and software are used for server and terminal processing: a communication interface for sending and receiving information over the network, an AI engine for image and emotion analysis, and a user interface for accepting user operations. Specifically, smart glasses, tablet devices, and cloud-based servers are used.

[1135] Specific examples

[1136] For example, when a user purchases "cabbage, carrots, and chicken breast" in a store and begins cooking, the following prompt sentence is used:

[1137] Recipe suggestion prompt: "Suggest recipes based on user data and emotional state."

[1138] Sentiment analysis prompt: "Please analyze the user's emotions from this cooking face image."

[1139] Social media caption generator prompt: "Generate a positive caption based on a photo of food and your current emotional state."

[1140] This system will enable users to have a cooking experience that is tailored to their individual emotional state, and is expected to make the cooking process in physical stores even more enjoyable and stress-free.

[1141] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1142] Step 1:

[1143] Receiving purchasing information and managing food inventory

[1144] When a user purchases ingredients at the store's automatic scanning register, the terminal receives the purchase information and sends it to the cloud server via the network. The input is purchase information such as cabbage, carrots, and chicken breast, and the server stores this information in a database. This data processing ensures that the user's ingredient inventory is recorded in the database in an up-to-date state.

[1145] Step 2:

[1146] Emotion analysis

[1147] Before the user starts cooking, the device's camera takes a photo of the user's face. The input is the user's facial image, which the device sends to a cloud server. The server uses an emotion engine to analyze the user's current emotional state and sends the results back to the device. This data calculation outputs an emotional state such as "tired" or "relaxed." An example prompt is: "Please analyze the user's emotions from this facial image while cooking."

[1148] Step 3:

[1149] Recipe suggestions

[1150] The server generates an optimal recipe based on inventory information, past purchase history, and the emotional state obtained in step 2. The input is the ingredient inventory, purchase history, and emotional state, and the output is a suggested optimal recipe. The generated recipe is sent to the terminal and displayed so that the user can select it. Example prompt: "Please suggest a recommended recipe based on the user data and emotional state."

[1151] Step 4:

[1152] Cooking assistance

[1153] When the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking progress with an AI camera and provides appropriate audio and video guidance in real time. It also uses an emotion engine to analyze the user's emotional state and provide encouraging words if the user is feeling stressed. The input is video and audio of the user cooking, and the output is a guide message and encouraging message.

[1154] Step 5:

[1155] Support for taking photos of the finished dishes and posting them on social media

[1156] Once the dish is ready, the user takes a photo of the dish using their device and uploads it to the server. The input is the photo of the dish, and the server uses this to generate a caption for posting on social media. The generated caption and photo are sent back to the device for the user to review and edit, and then post to social media. The output is the generated caption and photo. Example prompt: "Generate a positive caption based on the photo of the dish and your current emotional state."

[1157] In this way, a system has been constructed in which appropriate data calculations and processing are performed based on input data at each step, and the output is provided to the user.

[1158] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1159] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1160] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1161] [Fourth embodiment]

[1162] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1163] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1164] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1165] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1166] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1167] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1168] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1169] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1170] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1171] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1172] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1173] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1174] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1175] The system of this invention aims to provide users with an efficient and enjoyable cooking experience through the following processing steps: The entire system is designed to receive information on ingredients purchased online by users, suggest optimal recipes using AI, provide real-time support during the cooking process, and allow users to enjoyably share the finished dish on social media.

[1176] Ingredient purchase and registration

[1177] When a user purchases ingredients on an online shopping site, a list of ingredients is generated. The device receives this information and sends it to the server. The server stores the received ingredient information in a database and manages ingredient inventory. For example, if a user purchases cabbage, carrots, and chicken breast, these ingredients are registered in the database.

[1178] Recipe suggestions

[1179] The server uses AI to generate optimal recipes based on the user's ingredient inventory and past purchase history. The generated recipes also take into account the user's past ratings and preferences, as well as the expiration dates of ingredients. The generated recipes are sent from the server to the user's device, and the user can select from a list of suggested recipes. For example, a "stir-fried chicken breast" using chicken breast, cabbage, and carrots may be suggested.

[1180] Cooking support

[1181] When the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking progress with an AI camera and provides appropriate audio and video guidance in real time. This allows the user to proceed step by step with specific cooking procedures, allowing even beginners to enjoy cooking with confidence. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided via audio and video.

[1182] Support for posting cooking results on social media

[1183] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to a server, where AI generates a caption for posting on social media. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media. For example, a photo of a completed "Stir-fried chicken breast" could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[1184] System benefits

[1185] This system allows users to easily purchase ingredients, reduce waste, and enjoy a healthy diet. AI support makes cooking easy even for beginners, reducing stress during cooking and increasing the enjoyment of cooking. In addition, support for posting to social media allows users to easily share their cooking results, increasing their motivation. Specifically, the system is expected to enrich users' lives by supporting a series of processes, including ingredient management, recipe suggestions, cooking support, and social media posting.

[1186] The processing flow will be explained below.

[1187] Ingredient purchase and registration

[1188] Step 1:

[1189] A user purchases groceries from an online shopping site.

[1190] Step 2:

[1191] The device generates a list of purchased ingredients (e.g., cabbage, carrots, chicken breast) and sends it to the server.

[1192] Step 3:

[1193] The server stores the received ingredient information in a database and updates ingredient inventory information.

[1194] Recipe suggestions

[1195] Step 1:

[1196] The server accesses the database to obtain the user's food inventory and purchase history.

[1197] Step 2:

[1198] The server uses AI to analyze the acquired ingredients and historical data and generate the optimal recipe for the user.

[1199] Step 3:

[1200] The server sends the generated recipe to the user's terminal.

[1201] Cooking support

[1202] Step 1:

[1203] The user selects a suggested recipe.

[1204] Step 2:

[1205] The device will instruct you to start cooking and display a list of the necessary preparations.

[1206] Step 3:

[1207] The device uses an AI camera to monitor the user's cooking progress and provides real-time audio and video guidance for each cooking step.

[1208] Support for posting cooking results on social media

[1209] Step 1:

[1210] The user takes a photo of the finished dish using the device.

[1211] Step 2:

[1212] The device sends the photograph to the server.

[1213] Step 3:

[1214] The server uses AI to generate appropriate captions for the photos and send them to the user's device.

[1215] Step 4:

[1216] The device displays the generated caption and photo on a screen for posting to SNS, and the user can confirm and edit the post before posting it to SNS.

[1217] Example 1

[1218] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1219] In modern life, many people find it difficult to maintain a healthy diet in their busy daily lives. Beginners in cooking also feel anxious and stressed during the cooking process, making it difficult to smoothly navigate the entire process from selecting the right ingredients, creating recipes, and even sharing photos of the food after cooking. Furthermore, there is a need for a way to effectively use purchased ingredients without wasting them. A system is needed to solve these challenges and provide an efficient and enjoyable cooking experience.

[1220] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1221] In this invention, the server includes means for receiving purchase information, means for storing ingredient inventory in a database based on the purchase information, a generative AI model means for generating optimal recipes based on ingredient inventory and purchase history, means for sending the generated recipes to a user terminal, means for providing audio and video guidance to support the user in cooking in real time, a generative AI model means for receiving photos of the completed dish and generating captions for posting to an SNS, and means for displaying on the user terminal a preparation screen for posting the generated captions and photos to an SNS. This allows users to smoothly and efficiently perform a series of processes from ingredient management to recipe suggestions, cooking support, and sharing on an SNS.

[1222] The "means for receiving purchase information" is a device or program for acquiring information about ingredients purchased online by a user and importing it into the system.

[1223] The "means for storing ingredient inventory in a database based on purchase information" refers to a device or program for analyzing the received purchase information and registering and managing ingredient inventory information in a database.

[1224] The "generative AI model means for generating optimal recipes based on ingredient inventory and purchase history" is a module that uses artificial intelligence to automatically create recipes that meet the user's needs, using the user's ingredient inventory and past purchase history.

[1225] "Means for transmitting the generated recipe to the user terminal" refers to a device or program for transmitting the recipe generated by the AI ​​model to the terminal used by the user.

[1226] "Means for providing audio and video guidance to assist users in cooking in real time" refers to a device or program that provides audio and video guidance at appropriate times while the user is cooking, thereby assisting the cooking process.

[1227] The "generative AI model means for receiving a photo of a finished dish and generating a caption for posting on social media" is a module that utilizes artificial intelligence to receive a photo of a dish taken by a user and, based on the photo, create an appropriate caption for posting on social media.

[1228] "Means for displaying on a user's device a screen for preparing to post the generated captions and photos to a social networking site" refers to a device or program that displays on a user's device a screen that allows the user to review and edit the captions and food photos generated by the AI ​​and post them to a social networking site.

[1229] An "AI camera that monitors cooking status" is a device that combines a camera and artificial intelligence to automatically monitor the user's actions and status while cooking, and provide feedback and guidance as needed.

[1230] The "generative AI model algorithm that takes user feedback into account to improve recipes" is an algorithm that uses artificial intelligence to reflect user ratings and comments and continuously optimize the recipe suggestions.

[1231] The system of the present invention aims to provide users with an efficient and enjoyable cooking experience. The main components of the system include a server, a terminal, and an AI model.

[1232] Overall system overview

[1233] This system is designed to receive information about ingredients purchased online by users, suggest optimal recipes using a generative AI model, provide real-time support during the cooking process, and allow users to enjoy sharing the finished dish on social media. Specifically, it consists of the following steps:

[1234] Ingredient purchase and registration

[1235] When a user purchases ingredients on an online shopping site, a list of ingredients is generated. For example, if a user purchases cabbage, carrots, and chicken breasts, the information is received by the terminal and sent to the server. The server stores the received ingredient information in a database and runs a program to manage the ingredient inventory.

[1236] Recipe suggestions

[1237] The server uses a generative AI model to generate optimal recipes based on the user's ingredient inventory and past purchase history. For example, a prompt such as "Please suggest the best recipe using cabbage, carrots, and chicken breast" is input into the generative AI model. The generated recipe also takes into account the user's preferences and the expiration dates of the ingredients. The generated recipe is sent from the server to the user's device, and the user can select the suggested recipe.

[1238] Cooking support

[1239] Once the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking process with an AI camera and provides appropriate audio and video guidance in real time. This allows the user to take specific cooking steps step by step. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided via audio and video.

[1240] Support for posting cooking results on social media

[1241] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to a server, where the generative AI model generates a caption for posting on social media. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media. For example, a photo of a dish made with cabbage, carrots, and chicken breast could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[1242] Hardware and software used

[1243] The main hardware used in this system includes the device operated by the user (smartphone, tablet, PC, etc.) and the AI ​​camera that monitors the cooking status, while the main software includes a database management system running on the server, a generative AI model, a recipe generation program, and audio and video guide programs.

[1244] Specific prompt examples

[1245] An example of a prompt sentence to input to the generative AI model is as follows:

[1246] "Please suggest the best recipes using the cabbage, carrots, and chicken breast that the user has purchased. Also, please guide the user through the cooking steps of the suggested recipes in real time with audio and video guides. Furthermore, please generate captions for posting on social media based on photos of the finished dishes."

[1247] This allows the system of the present invention to help users smoothly go through a series of processes, from managing ingredients to suggesting recipes, supporting cooking, and sharing on social media.

[1248] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1249] Step 1: Receive purchase information

[1250] When a user purchases ingredients from an online shopping site, a purchase completion screen is displayed. This purchase completion screen includes a list of ingredients purchased by the user. The device extracts necessary information from this screen and sends it to the server as purchase information.

[1251] Specific operation: The user completes a purchase at the online store. The device automatically extracts the ingredient list from the purchase completion screen. The device then sends the ingredient information to the server.

[1252] Input: URL and HTML content of the purchase completion screen

[1253] Output: Extracted ingredients list (e.g. cabbage, carrot, chicken breast)

[1254] Step 2: Registering ingredient data

[1255] The server analyzes the received ingredient information and registers the list of ingredients in a database. This allows ingredient inventory information to be managed for each user. For example, information on cabbage, carrots, and chicken breast is registered.

[1256] Specific operation: The server analyzes the received ingredient information and converts it into an appropriate data format. The server then registers the ingredient inventory information in the database.

[1257] Input: List of ingredients (e.g. cabbage, carrots, chicken breast)

[1258] Output: Updated database inventory information

[1259] Step 3: Recipe generation and suggestions

[1260] The server retrieves ingredient inventory information from the database and analyzes the user's past purchase history and ratings. It then uses a generative AI model to generate an optimal recipe. For example, a prompt such as "Please suggest a recipe using cabbage, carrots, and chicken breast" is input into the generative AI model. The generated recipe is then sent from the server to the user's device.

[1261] Specific operation: The server retrieves ingredient information and past purchase history from the database. It inputs prompts into the generative AI model to generate an optimal recipe. The server then sends the generated recipe to the user's device.

[1262] Input: Ingredient inventory information, past purchase history, user ratings

[1263] Output: Suggested recipe

[1264] Step 4: Cooking support

[1265] Once the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking process with an AI camera and provides real-time audio and video instructions. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided.

[1266] Specific operation: The user selects a suggested recipe on the device. The device starts audio and video guidance. The AI ​​camera monitors the user's cooking actions and provides feedback as needed.

[1267] Input: Suggested recipes, real-time camera footage

[1268] Output: Audio guide, video guide, feedback information

[1269] Step 5: Support for posting cooking results on social media

[1270] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to the server, where the generative AI model generates a caption for posting on social media. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media.

[1271] Specific operations: A user takes a photo of a dish on their device and uploads it. The device sends the photo to the server. The generative AI model generates a caption. The server sends the generated caption to the device. The user checks and edits the caption and posts it to social media.

[1272] Input: Food photo

[1273] Output: Caption for SNS posting, SNS posting preparation screen

[1274] (Application example 1)

[1275] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1276] Conventional food ingredient management and recipe suggestion systems make it difficult to use purchased ingredients efficiently and provide insufficient support during cooking, so they are unable to completely eliminate the hassle and stress that users experience when cooking. Furthermore, sharing completed dishes on social media is a self-resolved process, so support is needed to further increase user satisfaction.

[1277] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1278] In this invention, the server includes means for receiving purchase information, means for storing ingredient inventory in a database based on the purchase information, AI means for generating optimal recipes based on ingredient inventory and purchase history, means for transmitting the generated recipes to a user terminal, means for providing audio and video guidance to support the user in cooking in real time, AI means for receiving photos of the completed dish and generating captions for posting to SNS, means for displaying on the user terminal a preparation screen for posting the generated caption and photos to SNS, and AI means for collecting information in real time as the user progresses with cooking and providing individual advice. This not only allows users to use purchased ingredients efficiently and without waste, but also reduces stress by receiving support during cooking and allows them to easily share their completed dishes on SNS.

[1279] The "means for receiving purchase information" is a function for obtaining data on ingredients and products purchased by users from online shopping sites and other digital platforms and delivering that information to the processing system.

[1280] "Means for storing food ingredient inventory in a database based on purchase information" is a system for recording and managing the inventory status of ingredients and products held by the user in a digital database based on the received purchase information.

[1281] "AI method for generating optimal recipes based on ingredient inventory and purchase history" is a technology that uses artificial intelligence to generate and suggest optimal cooking recipes for users based on ingredient inventory information stored in a database and the user's past purchase history.

[1282] "Means for sending the generated recipe to the user's device" refers to a communication function for sending recipe information generated by AI to devices such as smartphones, tablets, and PCs used by users.

[1283] "Means for providing audio and video guidance to assist users in cooking in real time" refers to a system that provides audio and video instructions and guidance in real time at appropriate times as the user proceeds with cooking, thereby assisting the user in cooking.

[1284] "AI means for receiving photos of completed dishes and generating captions for posting on social media" is a function that receives photos of dishes taken by users and uses artificial intelligence to generate appropriate captions for posting on social media based on those photos.

[1285] "Means for displaying a preparation screen on the user's device for posting the generated caption and photo to SNS" is a function that combines the caption generated by AI with the photo provided by the user, and displays a preparation screen on the device that the user can check and edit before posting to SNS.

[1286] "AI means for collecting information in real time as the user cooks and providing individualized advice" refers to artificial intelligence technology that monitors the user's cooking progress in real time and provides individualized advice and instructions to the user based on that data.

[1287] The system of the present invention receives information about ingredients purchased by a user and includes a series of procedures for supporting efficient cooking based on that information. The operation of the system will be described in detail below.

[1288] First, when a user purchases ingredients from an online shopping site, the purchase information is automatically sent to the server. The server uses a means for receiving the purchase information to store the information in a database. The stored data is used in subsequent processing.

[1289] Next, AI is used to generate optimal recipes based on the ingredients' inventory and purchasing history. This AI analyzes the information stored in the database and the user's past purchasing history to suggest optimal recipes. For this purpose, a generative AI model (e.g., GPT-3 or BERT) can be used.

[1290] The generated recipes are sent from the server to the user's device, which can include a smartphone or tablet, where the user can check the suggested recipes.

[1291] Once the user selects a suggested recipe, audio and video guidance is provided to assist with cooking in real time, allowing users to receive the necessary instructions at the appropriate time while cooking. The audio and video guidance can be realized using, for example, Google Cloud Text-to-Speech or AWS Polly.

[1292] Furthermore, it also includes a means to collect information in real time as the user cooks and provide personalized advice. This involves using AI cameras such as Raspberry Pi and Google Cloud Vision to monitor the cooking process. Based on this data, specific advice is provided to the user.

[1293] Once the dish is ready, the user uploads a photo to their device. The device then sends the photo to a server, where AI generates a caption for posting on social media. The generated caption and photo are displayed on the user's device, and the user can easily post it to social media after checking and editing. Social media integration can be achieved using, for example, the Twitter API or Facebook Graph API.

[1294] As a concrete example, consider the case where a user purchases chicken breast, cabbage, and carrots online. This information is stored in a database, and the AI ​​generates a recipe based on the prompt, "Please tell me an easy and delicious recipe using chicken breast, cabbage, and carrots." This generated recipe is sent to the user's device, and the user can proceed with cooking by following the instructions.

[1295] This system allows users to use purchased ingredients efficiently and without waste. In addition, real-time support and support for posting on social media make cooking more fun and easier to share.

[1296] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1297] Step 1:

[1298] A user purchases ingredients from an online shopping site. The terminal acquires this purchase information and sends it to a server. The input is the purchase information from the online shopping site, and the output is the purchase information sent to the server. Specifically, the terminal collects a list of ingredients purchased by the user (e.g., chicken breast, cabbage, and carrots) and sends it to the server.

[1299] Step 2:

[1300] The server stores the received purchase information in a database. The input is the purchase information sent from the device, and the output is the ingredient information stored in the database. The server uses, for example, Firebase to record the purchased ingredient information in the database.

[1301] Step 3:

[1302] The server uses AI to generate optimal recipes based on ingredient information in the database and past purchase history. The input is ingredient information in the database and past purchase history, and the output is the generated recipe. Specifically, the server uses a generative AI model (e.g., GPT-3) to generate a prompt such as, "Please tell me an easy and delicious recipe using chicken breast, cabbage, and carrots," and creates the optimal recipe based on that.

[1303] Step 4:

[1304] The server sends the generated recipe to the user's terminal. The input is the generated recipe, and the output is the recipe information sent to the user's terminal. The server uses a communication protocol to send the generated recipe (e.g., "Stir-fried chicken breast with cabbage and carrots") to the user's smartphone.

[1305] Step 5:

[1306] When a user selects a recipe, the device guides the user to start cooking. The input is the user's selection, and the output is the cooking start guidance. Specifically, the device displays audio and video guides and provides the user with specific instructions, such as "Cut the chicken breast into bite-sized pieces."

[1307] Step 6:

[1308] The device uses an AI camera to monitor the cooking process in real time to provide real-time support. The input is video information during cooking, and the output is real-time advice and instructions. The device uses, for example, Raspberry Pi and Google Cloud Vision to analyze the cooking process and provide voice guidance to the user on the next steps.

[1309] Step 7:

[1310] When the food is ready, the user takes a photo of it and uploads it to the device. The input is a photo of the food, and the output is the photo data sent to the server. The user takes a photo with their smartphone and uploads it to the device.

[1311] Step 8:

[1312] The server generates a caption for posting to social media based on the received photo. The input is a photo of the dish, and the output is the generated caption. Specifically, the server uses a generative AI model to generate the caption, "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[1313] Step 9:

[1314] The server sends the generated caption and photo to the user's device and displays a preparation screen for posting to SNS. The input is the generated caption and a photo of the dish, and the output is the preparation screen. The user can then check and edit the caption and post it to SNS.

[1315] Through these processing steps, users can use purchased ingredients efficiently and without waste. In addition, real-time support and support for posting on social media make cooking more enjoyable and reduce the effort required for sharing.

[1316] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1317] The system of this invention is designed to provide users with an efficient and enjoyable cooking experience. By combining the system with an emotion engine, it is possible to take into account the user's emotions and moods and provide more personalized assistance. The entire system is designed to receive information on ingredients purchased online by the user, use AI to suggest optimal recipes, provide real-time assistance with the cooking process, and allow users to enjoy sharing the finished dish on social media.

[1318] Ingredient purchase and registration

[1319] When a user purchases ingredients on an online shopping site, a list of ingredients is generated. The device receives this information and sends it to the server. The server stores the received ingredient information in a database and manages ingredient inventory. For example, if a user purchases cabbage, carrots, and chicken breast, these ingredients are registered in the database.

[1320] Recipe suggestions

[1321] The server uses AI to generate optimal recipes based on the user's ingredient inventory and past purchase history. The generated recipes also take into account the user's past ratings and preferences, as well as the expiration dates of ingredients. Furthermore, an emotion engine is used to analyze the user's current emotional state and suggest recipes that suit that state. The generated recipes are sent from the server to the user's device, and the user can select from a list of suggested recipes. For example, if the user is feeling a bit tired, a quick and easy "stir-fried chicken breast" may be suggested.

[1322] Cooking support

[1323] Once the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking progress with an AI camera and provides appropriate audio and video guidance in real time. It uses an emotion engine to recognize the user's emotions while cooking, and adjusts the content of the audio guidance, such as providing words of encouragement, if the user feels stressed. This allows the user to proceed step by step with specific cooking procedures, allowing even beginners to enjoy cooking with confidence. For example, instructions such as "Cut the chicken breast into bite-sized pieces" are provided via audio and video.

[1324] Support for posting cooking results on social media

[1325] Once the dish is complete, the user takes a photo and uploads it to their device. The device then sends the photo to a server, where AI generates a caption to post on social media. The caption is generated taking into account the user's emotional state, so if the user is feeling happy, a positive message such as "It turned out delicious!" will be added. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it on social media. For example, a photo of a completed "Stir-fried chicken breast" could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[1326] System benefits

[1327] This system allows users to easily purchase ingredients, reduce waste, and enjoy a healthy diet. The introduction of an emotion engine allows the system to take the user's emotions into consideration during cooking and when suggesting recipes, providing personalized support tailored to each individual user. This reduces the stress users feel during the cooking process and makes cooking even more enjoyable. In addition, support for posting to social media allows users to easily share the results of their cooking, increasing their motivation. Specifically, the system is expected to enrich users' lives by supporting a series of processes, including ingredient management, recipe suggestions, cooking support, and social media posting.

[1328] The processing flow will be explained below.

[1329] Ingredient purchase and registration

[1330] Step 1:

[1331] A user purchases groceries from an online shopping site.

[1332] Step 2:

[1333] The device generates a list of purchased ingredients (e.g., cabbage, carrots, chicken breast) and sends it to the server.

[1334] Step 3:

[1335] The server stores the received ingredient information in a database and updates ingredient inventory information.

[1336] Recipe suggestions

[1337] Step 1:

[1338] The server accesses the database to obtain the user's food inventory and purchase history.

[1339] Step 2:

[1340] The server uses AI to analyze the acquired ingredients and historical data and generate the optimal recipe for the user.

[1341] Step 3:

[1342] The server uses an emotion engine to recognize the user's current emotional state, which is collected through the device's camera and voice input device.

[1343] Step 4:

[1344] The server suggests recipes based on the user's emotional state. For example, a tired user might be recommended a simple dish (e.g., "stir-fried chicken breast").

[1345] Step 5:

[1346] The server sends the generated recipe to the user's terminal.

[1347] Cooking support

[1348] Step 1:

[1349] The user selects a suggested recipe.

[1350] Step 2:

[1351] The device will instruct you to start cooking and display a list of the necessary preparations.

[1352] Step 3:

[1353] The device uses an AI camera to monitor the user's cooking progress.

[1354] Step 4:

[1355] The device provides real-time audio and video instructions for each cooking step, such as "Cut the chicken breast into bite-sized pieces."

[1356] Step 5:

[1357] The device uses an emotion engine to recognize the user's emotions while cooking, and if the user is feeling stressed, it will offer words of encouragement or advice.

[1358] Support for posting cooking results on social media

[1359] Step 1:

[1360] The user takes a photo of the finished dish using the device.

[1361] Step 2:

[1362] The device sends the photograph to the server.

[1363] Step 3:

[1364] The server uses AI to generate appropriate captions for photos, taking into account the user's emotional state.

[1365] Step 4:

[1366] The server sends the generated caption and photo to the user terminal.

[1367] Step 5:

[1368] The device displays the generated caption and photo on a screen for posting to SNS, and the user can confirm and edit the post before posting it to SNS.

[1369] Specific examples of programs

[1370] Example: Purchasing ingredients and suggesting recipes

[1371] A user purchases cabbage, carrots, and chicken breasts from an online shopping site.

[1372] The terminal transmits this ingredient information to the server.

[1373] The server registers cabbage, carrots, and chicken breasts in the database.

[1374] The server uses AI and an emotion engine to generate recipes based on the user's inventory information and purchase history. For example, if the user is tired, it will suggest "easy stir-fried chicken breast."

[1375] The server transmits the generated recipe to the user terminal, and the user selects the recipe.

[1376] Example: Cooking support and social media posts

[1377] The user selects "Easy Stir-fry Chicken Breast" and starts cooking.

[1378] The device displays a list of necessary preparations and monitors the cooking progress using an AI camera.

[1379] The device provides audio and video guidance such as "Cut the chicken breast into bite-sized pieces."

[1380] Along the way, the device will recognize the user's emotions and, if they are feeling stressed, will offer words of encouragement such as, "It's okay, everything is going well!"

[1381] After cooking is complete, the user takes a photo of the finished dish, which the terminal then sends to the server.

[1382] The server uses AI to generate captions such as "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!" and sends them to the user's device.

[1383] Users can review and edit photos and captions and post them to social media.

[1384] Example 2

[1385] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1386] Until now, there has been no system that comprehensively supports the entire cooking process, from purchasing ingredients, inventory management, recipe suggestions, cooking assistance, and even posting to social media. Furthermore, technology that recognizes users' emotions during cooking and personalizes assistance in real time has been insufficient. As a result, users have had to perform numerous manual operations, which often leads to stress and difficulty during cooking. The present invention aims to solve these problems and provide users with a consistent, stress-free, and enjoyable cooking experience.

[1387] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1388] In this invention, the server includes: means for receiving purchase information; means for storing ingredient inventory in a database based on the purchase information; artificial intelligence means for generating optimal recipes based on ingredient inventory and purchase history; means for transmitting the generated recipes to a user terminal; means for providing audio and video guidance to support the user in cooking in real time; means including an emotion engine that recognizes the user's emotions while cooking and adjusts the support content based on the emotions; artificial intelligence means for receiving photos of the completed dish and generating captions to post to an SNS; and means for displaying on the user terminal a preparation screen for posting the generated caption and photo to an SNS. This enables users to manage ingredients, receive optimal recipe suggestions, receive real-time cooking support, receive support based on their emotions, and smoothly post to an SNS.

[1389] The "means for receiving purchase information" is a function for obtaining information about ingredients purchased by a user on an online shopping site.

[1390] The "means for storing ingredient inventory in a database based on purchase information" is a function for recording and managing ingredient inventory information in a database using the received purchase information.

[1391] "Artificial intelligence means for generating optimal recipes based on ingredient inventory and purchase history" is an AI technology that utilizes ingredient inventory information and past purchase history stored in a database to suggest optimal recipes to users.

[1392] "Means for sending the generated recipe to the user's terminal" is a function for sending recipe information generated by the AI ​​to the user's terminal.

[1393] The "means for providing audio and video guidance to assist the user in cooking in real time" is a function for providing audio and video instructions to the user in real time while cooking.

[1394] "Means including an emotion engine that recognizes the user's emotions while cooking and adjusts the support content based on those emotions" is a function that analyzes the user's emotional state and dynamically changes and provides the support content accordingly.

[1395] "Artificial intelligence means for receiving photos of completed dishes and generating captions for posting on social media" refers to AI technology that analyzes photos of dishes uploaded by users and generates accompanying captions for posting on social media.

[1396] "Means for displaying on the user's device a preparation screen for posting the generated caption and photo to SNS" is a function for displaying on the user's device a confirmation screen for posting the caption generated by AI and the photo uploaded by the user to SNS.

[1397] An "artificial intelligence camera" is a camera device that incorporates AI technology to monitor and analyze cooking conditions in real time.

[1398] "Artificial intelligence algorithms" are computational methods for optimizing and improving functionality based on user feedback and experience.

[1399] MODE FOR CARRYING OUT THE INVENTION

[1400] This invention is a system designed to provide users with an efficient and enjoyable cooking experience. Its key components are managing ingredient purchasing information, generating and suggesting optimal recipes, providing real-time cooking support, and supporting posting to social media. The system takes into account the user's emotional state to provide more personalized support.

[1401] Ingredient purchase and registration

[1402] When a user purchases ingredients from an online shopping site, the device receives the purchase information. The device converts this information into a standard format and sends it to the server. The server stores the received ingredient information in a database and manages inventory. For example, if a user purchases cabbage, carrots, and chicken breast, this information is registered in the database and managed as inventory.

[1403] Recipe suggestions

[1404] The server uses artificial intelligence to generate optimal recipes based on the user's ingredient inventory information and past purchase history stored in a database. It also uses an emotion engine to analyze the user's current emotional state and suggests recipes accordingly. The generated recipes are sent from the server to the user's device, and the user can select one of the suggested recipes. For example, if the user is feeling a bit tired, a quick and easy "stir-fried chicken breast" may be suggested.

[1405] Cooking support

[1406] Once the user selects a suggested recipe, the device will guide them to start cooking. The device uses an AI camera to monitor the cooking progress in real time and provides audio and video guidance. It also uses an emotion engine to recognize the user's emotions while cooking and adjusts the assistance as needed. For example, if the user is feeling stressed, the device will provide encouraging words. Specifically, it will provide audio and video instructions such as "Cut the chicken breast into bite-sized pieces."

[1407] Support for posting cooking results on social media

[1408] Once the dish is ready, the user takes a photo and uploads it to their device. The device then sends the photo to a server, which uses a generative AI model to generate a caption for the social media post. The caption is generated taking into account the user's emotional state. The generated caption is then sent from the server to the user's device, where the user can review and edit the photo and caption before posting it to social media. For example, a photo of a completed "Stir-fried chicken breast" could be captioned with "Stir-fried chicken breast with plenty of fresh cabbage and carrots is ready!"

[1409] Prompt Sentence Examples

[1410] The following is an example of a prompt sentence for generating a caption to post on social media. Assume that the emotional state is "joy," the dish is "stir-fried chicken breast," and the ingredients used are cabbage and carrots.

[1411] Generate a caption for a user to post on social media. The emotional state is joy. The dish is "Stir-fried chicken breast" and the ingredients are cabbage and carrots.

[1412] An example of the generated caption is:

[1413] "I finished this stir-fried chicken breast dish with lots of fresh cabbage and carrots! It was delicious!"

[1414] This system efficiently supports users through the entire process, from managing ingredients to cooking and posting on social media. In particular, the introduction of an emotion engine enables personalized support according to the user's individual state, further improving the cooking experience.

[1415] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1416] Specific explanation of system program processing

[1417] Ingredient purchase and registration

[1418] Step 1:

[1419] A user purchases groceries from an online shopping site.

[1420] Input: Information about ingredients purchased by the user

[1421] Output: Purchased ingredients list

[1422] Step 2:

[1423] The terminal receives the purchase information and converts it into a standard format, specifically the name, quantity, and purchase date of the purchased ingredients.

[1424] Input: Purchased food list

[1425] Output: Standardized grocery shopping list

[1426] Step 3:

[1427] The terminal transmits the list of purchased ingredients to the server.

[1428] Input: Standardized grocery shopping list

[1429] Output: A request to send data to the server

[1430] Step 4:

[1431] The server stores purchase information in a database and manages food inventory. For example, purchased cabbage, carrots, and chicken breasts are recorded in the database.

[1432] Input: Standardized grocery shopping list

[1433] Output: Food inventory database

[1434] Recipe suggestions

[1435] Step 1:

[1436] The server acquires the user's food inventory information and past purchase history from the database.

[1437] Input: User ID

[1438] Output: Food inventory information, purchase history

[1439] Step 2:

[1440] Based on the information retrieved by the server, artificial intelligence methods are used to generate optimal recipes. The AI ​​model also takes into account the expiration dates of ingredients and past user ratings.

[1441] Input: Ingredient inventory information, purchase history

[1442] Output: AI-generated recipe list

[1443] Step 3:

[1444] The server uses an emotion engine to obtain the user's current emotional state and suggests recipes that match it.

[1445] Input: User emotion data

[1446] Output: Personalized recipe

[1447] Step 4:

[1448] The server sends the generated recipe to the user's terminal.

[1449] Input: Personalized Recipe

[1450] Output: Recipe displayed on user's device

[1451] Cooking support

[1452] Step 1:

[1453] The user selects a suggested recipe.

[1454] Input: Recipe list

[1455] Output: Selected recipes

[1456] Step 2:

[1457] The terminal will guide the user to start cooking.

[1458] Input: Selected recipe

[1459] Output: Cooking start notification

[1460] Step 3:

[1461] The device uses an AI camera to monitor the cooking progress in real time.

[1462] Input: Real-time cooking footage

[1463] Output: Monitoring data

[1464] Step 4:

[1465] The device provides audio and video guidance, explaining specific cooking steps to the user. At the same time, it uses an emotion engine to recognize the user's emotions while cooking and provides appropriate guidance to reduce stress.

[1466] Input: Monitoring data, real-time emotion data

[1467] Output: Audio guide, video guide

[1468] Support for posting cooking results on social media

[1469] Step 1:

[1470] The user takes a photo of the finished dish and uploads it to the device.

[1471] Input: Food photo

[1472] Output: Uploaded photo data

[1473] Step 2:

[1474] The device sends the photo to a server, which uses a generative AI model to generate a caption for the social media post.

[1475] Input: Uploaded photo data

[1476] Output: Generated captions

[1477] Step 3:

[1478] The server transmits the generated captions to the user terminal.

[1479] Input: Generated caption

[1480] Output: Captions displayed on the user's device

[1481] Step 4:

[1482] Users can review and edit the caption and photo, then post it to social media.

[1483] Input: displayed caption, photo

[1484] Output: SNS post data

[1485] Prompt Sentence Examples

[1486] The following is an example of a prompt sentence for generating a caption to post on social media. Assume that the emotional state is "joy," the dish is "stir-fried chicken breast," and the ingredients used are cabbage and carrots.

[1487] Generate a caption for a user to post on social media. The emotional state is joy. The dish is "Stir-fried chicken breast" and the ingredients are cabbage and carrots.

[1488] An example of the generated caption is:

[1489] "I finished this stir-fried chicken breast dish with lots of fresh cabbage and carrots! It was delicious!"

[1490] (Application example 2)

[1491] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1492] Conventional cooking assistance systems lack personalized support based on the user's individual emotions and moods, making the cooking process monotonous. Furthermore, physical stores lack effective support methods to improve the in-store cooking experience. There is a need for a system that can resolve the stress and problems users experience while cooking in real time and provide a more enjoyable cooking experience.

[1493] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1494] In this invention, the server includes means for receiving purchase information, means for storing ingredient inventory in a database based on the purchase information, AI means for generating optimal recipes based on ingredient inventory and purchase history, means for transmitting the generated recipes to a user terminal, means for providing audio and video guidance to support the user in cooking in real time, AI means for receiving photos of the completed dish and generating captions for posting to SNS, means for displaying on the user terminal a preparation screen for posting the generated caption and photo to SNS, means for analyzing the user's emotional state using an emotion engine and providing personalized recipe suggestions and cooking support, and means for providing customers in the store with a personalized cooking experience and cooking support in a tasting corner. This makes it possible to provide personalized cooking support based on the user's emotional state and improve the cooking experience in a physical store.

[1495] The "means for receiving purchase information" is a means for receiving information about ingredients purchased by a user from an online shopping site or an automatic scanning register and inputting the information into the system.

[1496] The "means for storing food stock in a database based on purchase information" refers to a means for recording and managing the status of the user's food stock in a database based on the received purchase information.

[1497] "AI method for generating optimal recipes based on ingredient inventory and purchase history" refers to a method that uses ingredient inventory information and past purchase history to create recipes that are most suitable for the user using AI technology.

[1498] "Means for sending the generated recipe to the user's device" refers to means for sending the recipe generated by AI to the user's device such as a smartphone or tablet via communication.

[1499] "Means for providing audio and video guidance to assist users in cooking in real time" refers to means for providing necessary procedures and advice to users in the course of cooking in real time using audio and video.

[1500] "AI means for receiving a photo of a completed dish and generating a caption for posting on social media" refers to a means for receiving a photo of a completed dish and using AI to generate an appropriate caption for posting on social media.

[1501] "Means for displaying on a user's device a preparation screen for posting the generated caption and photo to an SNS" refers to means for displaying on a user's device a preparation screen for the user to check and edit the generated caption and photo and post to an SNS.

[1502] "Means for analyzing a user's emotional state using an emotion engine and providing personalized recipe suggestions and cooking assistance" refers to means for analyzing a user's current emotional state using an emotion engine and providing individually optimized recipe suggestions and cooking assistance based on that state.

[1503] "Means for providing in-store customers with a personalized cooking experience and providing cooking support in the tasting corner" refers to means for individually optimizing the cooking experience of customers in-store and providing cooking support in the tasting corner.

[1504] This invention is a system for improving customers' cooking experience in physical stores, and includes the following functions and processing flow.

[1505] Receiving purchasing information and managing food inventory

[1506] When a user purchases ingredients at the store's automatic scanning register, the device receives the purchase information and sends it to the cloud server via the network. The server stores the received purchase information in a database and manages the user's ingredient inventory. For example, if a user purchases cabbage, carrots, and chicken breast, the information about these ingredients is recorded in the database.

[1507] Recipe suggestions

[1508] The server uses AI technology to generate optimal recipes based on ingredient inventory information and past purchase history. It also utilizes an emotion engine to analyze the user's current emotional state and suggest personalized recipes based on that. For example, if the user is tired, simple and quick recipes will be suggested. These suggested recipes are then sent to the user's device, where they can be selected.

[1509] Cooking assistance

[1510] When the user selects a suggested recipe, the device will guide them to start cooking. During cooking, an AI camera monitors the cooking progress, and the device provides real-time audio and video guidance. The device also uses an emotion engine to analyze the user's emotional state, and if the user is feeling stressed, it will provide encouraging words as audio guidance. For example, encouraging messages such as "Keep up the good work, your delicious meal will be ready soon" are played.

[1511] Support for posting finished dishes on social media

[1512] Once the dish is ready, the user takes a photo of the dish using their device and uploads it to the server. The server then uses AI to generate a caption for posting on social media. This caption is generated taking into account the user's emotional state, so it contains a positive message. For example, a caption might be generated such as, "Delicious chicken breast stir-fry with fresh cabbage and carrots is ready!" The generated caption and photo are displayed on the user's device, where the user can review and edit it before posting it to social media.

[1513] Program processing

[1514] The following hardware and software are used for server and terminal processing: a communication interface for sending and receiving information over the network, an AI engine for image and emotion analysis, and a user interface for accepting user operations. Specifically, smart glasses, tablet devices, and cloud-based servers are used.

[1515] Specific examples

[1516] For example, when a user purchases "cabbage, carrots, and chicken breast" in a store and begins cooking, the following prompt sentence is used:

[1517] Recipe suggestion prompt: "Suggest recipes based on user data and emotional state."

[1518] Sentiment analysis prompt: "Please analyze the user's emotions from this cooking face image."

[1519] Social media caption generator prompt: "Generate a positive caption based on a photo of food and your current emotional state."

[1520] This system will enable users to have a cooking experience that is tailored to their individual emotional state, and is expected to make the cooking process in physical stores even more enjoyable and stress-free.

[1521] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1522] Step 1:

[1523] Receiving purchasing information and managing food inventory

[1524] When a user purchases ingredients at the store's automatic scanning register, the terminal receives the purchase information and sends it to the cloud server via the network. The input is purchase information such as cabbage, carrots, and chicken breast, and the server stores this information in a database. This data processing ensures that the user's ingredient inventory is recorded in the database in an up-to-date state.

[1525] Step 2:

[1526] Emotion analysis

[1527] Before the user starts cooking, the device's camera takes a photo of the user's face. The input is the user's facial image, which the device sends to a cloud server. The server uses an emotion engine to analyze the user's current emotional state and sends the results back to the device. This data calculation outputs an emotional state such as "tired" or "relaxed." An example prompt is: "Please analyze the user's emotions from this facial image while cooking."

[1528] Step 3:

[1529] Recipe suggestions

[1530] The server generates an optimal recipe based on inventory information, past purchase history, and the emotional state obtained in step 2. The input is the ingredient inventory, purchase history, and emotional state, and the output is a suggested optimal recipe. The generated recipe is sent to the terminal and displayed so that the user can select it. Example prompt: "Please suggest a recommended recipe based on the user data and emotional state."

[1531] Step 4:

[1532] Cooking assistance

[1533] When the user selects a suggested recipe, the device will guide them to start cooking. The device monitors the cooking progress with an AI camera and provides appropriate audio and video guidance in real time. It also uses an emotion engine to analyze the user's emotional state and provide encouraging words if the user is feeling stressed. The input is video and audio of the user cooking, and the output is a guide message and encouraging message.

[1534] Step 5:

[1535] Support for taking photos of the finished dishes and posting them on social media

[1536] Once the dish is ready, the user takes a photo of the dish using their device and uploads it to the server. The input is the photo of the dish, and the server uses this to generate a caption for posting on social media. The generated caption and photo are sent back to the device for the user to review and edit, and then post to social media. The output is the generated caption and photo. Example prompt: "Generate a positive caption based on the photo of the dish and your current emotional state."

[1537] In this way, a system has been constructed in which appropriate data calculations and processing are performed based on input data at each step, and the output is provided to the user.

[1538] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1539] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1540] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1541] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1542] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1543] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1544] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1545] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1546] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1547] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1548] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1549] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1550] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1551] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1552] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1553] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1554] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1555] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1556] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1557] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1558] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1559] The following is further disclosed regarding the above embodiment.

[1560] (Claim 1)

[1561] means for receiving purchase information;

[1562] a means for storing food inventory in a database based on the purchasing information;

[1563] AI means to generate optimal recipes based on ingredient inventory and purchasing history;

[1564] means for transmitting the generated recipe to a user terminal;

[1565] means for providing audio and video guidance to assist a user in cooking in real time;

[1566] An AI method that receives photos of the finished dish and generates captions for posting on social media;

[1567] A means for displaying a preparation screen for posting the generated caption and photo to an SNS on a user terminal;

[1568] A system including:

[1569] (Claim 2)

[1570] The system according to claim 1, characterized in that it is equipped with an AI camera that monitors the cooking status.

[1571] (Claim 3)

[1572] 10. The system of claim 1, including an AI algorithm that takes user feedback into account to improve the recipe.

[1573] "Example 1"

[1574] (Claim 1)

[1575] means for receiving purchase information;

[1576] a means for storing food inventory in a database based on the purchasing information;

[1577] A generative AI model means for generating optimal recipes based on ingredient inventory and purchasing history;

[1578] means for transmitting the generated recipe to a user terminal;

[1579] means for providing audio and video guidance to assist a user in cooking in real time;

[1580] A generative AI model means for receiving a photo of the completed dish and generating a caption for posting on social media;

[1581] A means for displaying a preparation screen for posting the generated caption and photo to an SNS on a user terminal;

[1582] A system including:

[1583] (Claim 2)

[1584] The system according to claim 1, characterized in that it is equipped with an AI camera that monitors the cooking status.

[1585] (Claim 3)

[1586] 10. The system of claim 1, further comprising a generative AI model algorithm that takes user feedback into account to improve the recipe.

[1587] "Application Example 1"

[1588] (Claim 1)

[1589] means for receiving purchase information;

[1590] a means for storing food inventory in a database based on the purchasing information;

[1591] AI means to generate optimal recipes based on ingredient inventory and purchasing history;

[1592] means for transmitting the generated recipe to a user terminal;

[1593] means for providing audio and video guidance to assist a user in cooking in real time;

[1594] An AI method that receives photos of the finished dish and generates captions for posting on social media;

[1595] A means for displaying a preparation screen for posting the generated caption and photo to an SNS on a user terminal;

[1596] AI means to collect information in real time as the user cooks and provide personalized advice;

[1597] A system including:

[1598] (Claim 2)

[1599] The system according to claim 1, characterized in that it is equipped with an AI camera that monitors the cooking status.

[1600] (Claim 3)

[1601] 10. The system of claim 1, including an AI algorithm that takes user feedback into account to improve the recipe.

[1602] "Example 2: Combining Emotion Engines"

[1603] (Claim 1)

[1604] means for receiving purchase information;

[1605] a means for storing food inventory in a database based on the purchasing information;

[1606] artificial intelligence means to generate optimal recipes based on ingredient inventory and purchasing history;

[1607] means for transmitting the generated recipe to a user terminal;

[1608] means for providing audio and video guidance to assist a user in cooking in real time;

[1609] means including an emotion engine that recognizes the emotion of the user while cooking and adjusts the assistance content based on the emotion;

[1610] An artificial intelligence means for receiving a photo of the completed dish and generating a caption for posting on social media;

[1611] A means for displaying a preparation screen for posting the generated caption and photo to an SNS on a user terminal;

[1612] A system including:

[1613] (Claim 2)

[1614] The system according to claim 1, characterized in that it is equipped with an artificial intelligence camera that monitors the cooking situation.

[1615] (Claim 3)

[1616] 10. The system of claim 1, including an artificial intelligence algorithm that takes user feedback into account to improve the recipe.

[1617] "Application example 2 when combining emotion engines"

[1618] (Claim 1)

[1619] means for receiving purchase information;

[1620] a means for storing food inventory in a database based on the purchasing information;

[1621] AI means to generate optimal recipes based on ingredient inventory and purchasing history;

[1622] means for transmitting the generated recipe to a user terminal;

[1623] means for providing audio and video guidance to assist a user in cooking in real time;

[1624] An AI method that receives photos of the finished dish and generates captions for posting on social media;

[1625] A means for displaying a preparation screen for posting the generated caption and photo to an SNS on a user terminal;

[1626] A means for analyzing the emotional state of a user using an emotion engine and providing personalized recipe suggestions and cooking assistance;

[1627] A means to provide in-store customers with a personalized cooking experience and cooking assistance at the tasting area;

[1628] A system including:

[1629] (Claim 2)

[1630] The system according to claim 1, characterized in that it is equipped with an AI camera that monitors the cooking status.

[1631] (Claim 3)

[1632] 10. The system of claim 1, including an AI algorithm that takes user feedback into account to improve the recipe. [Explanation of symbols]

[1633] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving purchase information; a means for storing food inventory in a database based on the purchasing information; AI means to generate optimal recipes based on ingredient inventory and purchasing history; means for transmitting the generated recipe to a user terminal; means for providing audio and video guidance to assist the user in cooking in real time; An AI method that receives photos of the finished dish and generates captions for posting on social media; A means for displaying on a user terminal a preparation screen for posting the generated caption and photo to an SNS; A system including:

2. The system according to claim 1, characterized in that it is equipped with an AI camera that monitors the cooking status.

3. 10. The system of claim 1, including an AI algorithm that takes user feedback into account to improve the recipe.

Images