System

Abstract

A system is provided.SOLUTION: A cooking support system using interactive generative artificial intelligence, comprising: means for inputting user information and transmitting the user information to a server; means for causing generative artificial intelligence to generate a recipe based on a designated food material and providing the recipe to a user; means for providing a next procedure to the user by voice or text in real time; and means for acquiring professional technical information, optimizing the professional technical information, and providing the optimized professional technical information to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Recently, rising prices and the impact of COVID-19 have led to an increase in the number of people cooking at home. However, for those who lack confidence in their cooking skills or who wish to further improve their cooking skills, traditional recipe books and online recipes alone are insufficient. Another issue is the difficulty of checking recipes when your hands are full while cooking. Furthermore, creating health-conscious menus requires specialized knowledge, which is a burden for many people. To address these issues, there is a need for a cooking support system that can provide appropriate advice in real time through dialogue with users and incorporates professional techniques and health information. [Means for solving the problem]

[0005] The present invention provides a cooking assistance system that uses interactive generative AI. It includes a means for inputting user information and sending it to a server, and a means for saving the input user information. It also includes a means for having the generative AI generate a recipe based on specified ingredients and providing that recipe to the user. It also includes a means for providing the user with real-time instructions on how to proceed by voice or text, and a means for acquiring professional technical information, optimizing it, and providing it to the user. The system also includes a means for having the generative AI generate a healthy menu based on health-related information, and providing that menu to the user. This not only allows users to cook with peace of mind, but also helps them achieve a healthy diet.

[0006] "Interactive generative artificial intelligence" is artificial intelligence that has the ability to generate and provide information in real time through dialogue with users.

[0007] A "cooking support system" is a collection of hardware and software that supports users' cooking activities.

[0008] "User Information" refers to personal information including a User's name, age, food preferences, allergy information, health information, etc.

[0009] A "server" is a computer system that provides services to client terminals over a network.

[0010] "Ingredients" are the raw materials used to prepare a dish.

[0011] A "recipe" is a document or data that describes in detail how to prepare a dish.

[0012] "Professional technical information" refers to information about techniques and methods used by professionally skilled chefs.

[0013] "Health-related information" refers to data relating to an individual's health status and nutritional balance.

[0014] A "menu" is a plan of meal combinations for a specific period of time (e.g., a week).

[0015] "Input means" refers to a device or interface that allows a user to input information.

[0016] "Storage means" refers to a memory or database system for recording and storing input data.

[0017] "Providing means" refers to an interface or device for presenting the generated information and recipes to the user.

[0018] "Audio output means" refers to speakers or voice generation software that conveys information to the user as audio.

[0019] "Optimization" is the process of adjusting information and procedures to obtain optimal results for a particular purpose.

[0020] "Real time" refers to processing and responding in close accordance with actual time. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention provides a cooking assistance system that utilizes interactive generative artificial intelligence. Specific embodiments of the system will be described below.

[0043] System Overview

[0044] Users can interact with the system through an interface and receive various cooking assistance. The system primarily utilizes information collected from the server, terminals, and users to suggest recipes, provide real-time assistance with procedures, provide professional technical information, and suggest health-based menus.

[0045] Program processing overview

[0046] The system operates through the following steps: collecting user information, generating recipes, providing real-time assistance, providing professional technical information, and suggesting healthy meals.

[0047] Collection of User Information

[0048] 1. When a user uses the system for the first time, the terminal displays a screen for the user to enter their name, age, food preferences, allergy information, and health information.

[0049] 2. The user enters this information and presses the submit button.

[0050] 3. The device sends the entered information to the server.

[0051] 4. The server stores the received user information in a database.

[0052] 5. This allows the system to build a foundation for providing customized assistance to each user.

[0053] Providing cooking recipes

[0054] 1. If a user wants to know recipes that use a specific ingredient, they can request, for example, "Tell me recipes that use tomatoes."

[0055] 2. The device receives this request and sends it to the server.

[0056] 3. The server uses the AI ​​to generate the optimal recipe based on the user's profile information and request.

[0057] 4. The generation AI returns the generated recipe information to the server.

[0058] 5. The server sends the recipe information to the terminal, and the terminal displays the recipe to the user.

[0059] As a specific example, a recipe such as "tomato and basil pasta" is generated and provided to the user.

[0060] Real-time assistance with cooking procedures

[0061] 1. When the user actually starts cooking and wants to know the next steps, they can request, for example, "Tell me the next steps."

[0062] 2. The device sends this request to the server.

[0063] 3. The server has the generation AI generate the next steps and send those steps back to the server.

[0064] 4. The server sends the procedure information to the terminal, and the terminal provides the procedure to the user by voice or text.

[0065] For example, instructions such as "Please finely chop the tomatoes" are provided in real time.

[0066] Providing professional technical information

[0067] 1. If a user wants to know professional technical information about a particular recipe, they can request, for example, "Tell me the pro tips for this recipe."

[0068] 2. The device sends this request to the server.

[0069] 3. The server queries a professional technical information database and optimizes the information for the generating AI.

[0070] 4. The generating AI provides optimized technical information to the server.

[0071] 5. The server sends the technical information to the terminal, and the terminal displays or audibly explains the technical information to the user.

[0072] As a specific example, information such as "Professionals blanch tomatoes before chopping them to make them taste even better" is provided.

[0073] Healthy menu suggestions

[0074] 1. If a user wants to know about healthy meals, they can request, for example, "Tell me about healthy meals."

[0075] 2. The device sends this request to the server.

[0076] 3. The server uses the AI ​​to generate the optimal menu based on the user's health checkup data and nutritional balance information.

[0077] 4. The generation AI generates a menu that takes nutritional balance into consideration and sends it back to the server.

[0078] 5. The server sends the menu information to the terminal, and the terminal displays the menu to the user.

[0079] For example, a menu might include "Breakfast: oatmeal and fruit, Lunch: grilled salmon and salad, Dinner: chicken and vegetable soup."

[0080] In this way, a cooking assistance system utilizing interactive generative artificial intelligence can provide users with customized recipes and procedures, professional techniques, and healthy menus, providing practical and effective support.

[0081] The processing flow will be explained below.

[0082] Collection of User Information

[0083] Step 1:

[0084] The user launches the app for the first time and is presented with a screen to enter user information.

[0085] Step 2:

[0086] The device will present the user with fields to enter their name, age, food preferences, allergy information, and health information.

[0087] Step 3:

[0088] The user enters the required information and presses the send button.

[0089] Step 4:

[0090] The terminal transmits the input information to the server.

[0091] Step 5:

[0092] The server stores the received user information in a database.

[0093] Providing cooking recipes

[0094] Step 1:

[0095] The user voices or types in a request for a recipe using a specific ingredient (e.g., "Tell me some recipes using tomatoes").

[0096] Step 2:

[0097] The terminal transmits the input request to the server.

[0098] Step 3:

[0099] The server references the user's profile information from the database and sends the request to the generation artificial intelligence.

[0100] Step 4:

[0101] Generative AI generates the optimal recipe based on the user's profile and specified ingredients.

[0102] Step 5:

[0103] The server receives the generated recipe information and transmits it to the terminal.

[0104] Step 6:

[0105] The device displays the generated recipe (e.g., "Pasta with Tomato and Basil") to the user.

[0106] Real-time assistance with cooking procedures

[0107] Step 1:

[0108] If a user is cooking and wants to know the next step, they can send a request by voice or text (e.g., "Tell me the next step").

[0109] Step 2:

[0110] The device sends a request to the server.

[0111] Step 3:

[0112] The server issues instructions to the generation artificial intelligence to generate the next step.

[0113] Step 4:

[0114] Generative AI generates next step information (e.g., "Please finely chop the tomatoes").

[0115] Step 5:

[0116] The server receives the generated procedure information and transmits it to the terminal.

[0117] Step 6:

[0118] The device will provide the user with next steps via voice or text.

[0119] Step 7:

[0120] The user follows the instructions and sends the request again if the next step is required.

[0121] Providing professional technical information

[0122] Step 1:

[0123] A user submits a request for pro technical information on a recipe (e.g., "What are some pro tips for this recipe?").

[0124] Step 2:

[0125] The device sends a request to the server.

[0126] Step 3:

[0127] The server queries a professional technical information database and provides the information to the generating artificial intelligence.

[0128] Step 4:

[0129] Generative AI optimizes technical information and generates specific advice (e.g., "Professionals recommend blanching tomatoes before chopping them to make them taste even better").

[0130] Step 5:

[0131] The server receives the optimized technical information and sends it to the terminal.

[0132] Step 6:

[0133] The terminal displays or audibly explains technical information to the user.

[0134] Healthy menu suggestions

[0135] Step 1:

[0136] If a user wants to know about healthy meals, they send a request (e.g., "Tell me about healthy meals").

[0137] Step 2:

[0138] The device sends a request to the server.

[0139] Step 3:

[0140] The server retrieves the user's health information and nutritional balance information from the database and provides it to the generating artificial intelligence.

[0141] Step 4:

[0142] Generative artificial intelligence generates nutritionally balanced menus based on health information (e.g., "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup").

[0143] Step 5:

[0144] The server receives the generated menu information and transmits it to the terminal.

[0145] Step 6:

[0146] The terminal displays the menu information to the user.

[0147] The above are the specific processing steps of the cooking assistance system that uses interactive generative artificial intelligence.

[0148] Example 1

[0149] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0150] Current cooking assistance systems often fail to adequately address the individual needs and circumstances of users. Specifically, they lack customized recipes and assistance that reflect the user's health status, food preferences, allergy information, etc. They also lack real-time assistance needed during cooking and a means to easily obtain professional technical information. Furthermore, there is a lack of healthy menu suggestions based on the user's health information, so more personalized assistance is needed.

[0151] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0152] In this invention, the server includes means for inputting user information and transmitting it to the server, means for saving the input user information, means for having the AI ​​generate a recipe based on specified ingredients and providing the recipe to the user, means for providing the user with the next steps in real time by voice or text when the user actually starts cooking and requests the next steps, means for obtaining professional technical information for a specific recipe, optimizing it, and providing it to the user, and means for having the AI ​​generate a healthy menu based on health-related information and providing the menu to the user. This makes it possible to provide customized cooking support and real-time support that meets the individual needs of the user, as well as professional technical information and healthy menu suggestions.

[0153] "User information" is a general term for data related to individual needs and circumstances, such as a user's name, age, food preferences, allergy information, and health information.

[0154] A "server" is a computer system that receives and stores information sent by a user, sends requests to the generating artificial intelligence, processes the results, and provides them to the user.

[0155] A "terminal" is a device through which a user inputs information, makes requests, and receives and displays responses from a server.

[0156] "Generative AI" is an AI model that generates optimal recipes, procedures, technical information, and menus based on a user's requests and profile.

[0157] A "recipe" is a list of instructions or ingredients for cooking a particular dish.

[0158] "Real-time assistance" refers to assistance that provides the user with the next cooking steps on the spot as they proceed with their cooking.

[0159] "Professional technical information" is information about the techniques and tips of professional chefs.

[0160] A "healthy menu" refers to a meal plan that takes into consideration the user's health and nutritional balance.

[0161] The present invention is a cooking assistance system that uses interactive generative artificial intelligence to provide users with customized cooking recipes, cooking procedures, professional technical information, and healthy menus. Specific embodiments of the present invention are described below.

[0162] The system mainly consists of a server, a terminal, and a user. The server collects user information, stores and processes data, and runs generative AI models. The terminal receives input from the user, communicates with the server, and displays information to the user.

[0163] Hardware and software used

[0164] Hardware:

[0165] Server: A high-performance computer (e.g., a cloud service server)

[0166] Devices: smartphones, tablets, computers

[0167] software:

[0168] Generative AI models: such as OpenAI's GPT-3

[0169] Database: MySQL, MongoDB

[0170] Communication method: REST API, WebSocket

[0171] Front-end technologies: HTML, JavaScript, Swift

[0172] Collection of User Information

[0173] When the user first starts the app, they enter their name, age, food preferences, allergy information, and health information on the input screen on their device. The entered information is sent from the device to the server and stored in a database on the server. This creates a foundation for providing support customized for each user.

[0174] Providing cooking recipes

[0175] If a user wants to know about recipes that use a specific ingredient, they can make a request, for example, "Tell me recipes that use tomatoes." This request is sent from the device to the server, which then uses the user's profile information and the request to generate the optimal recipe for the generative AI model. The generative AI model then sends the generated recipe information back to the server, which then sends it to the device and displays it to the user.

[0176] Example: "Tomato and basil pasta"

[0177] Real-time assistance with cooking procedures

[0178] When a user actually starts cooking and wants to know the next step, they can make a request, for example, "Tell me the next step." This request is sent from the device to the server, and the server has the generative AI model generate the next step and send the result back to the server. The server then sends the step information to the device, and the device provides the step to the user via voice or text.

[0179] Example: "Please chop the tomatoes finely."

[0180] Providing professional technical information

[0181] If a user wants to know professional technical information for a particular recipe, they can make a request, for example, "Tell me the professional tips for this recipe." This request is sent from the device to the server, which queries a database of professional technical information and optimizes that information for the generative AI model. The optimized technical information is then sent from the server to the device, which then provides that information to the user.

[0182] Example: "Professionals blanch tomatoes before chopping them to enhance their flavor."

[0183] Healthy menu suggestions

[0184] If a user wants to know about healthy menus, they can make a request, for example, "Tell me about healthy menus." This request is sent from the device to the server, and the server uses the user's health checkup data and nutritional balance information to generate an optimal menu using the generative AI model. The generated menu is then sent to the device via the server and displayed to the user.

[0185] Example: "Breakfast: Oatmeal and fruit, Lunch: Grilled salmon and salad, Dinner: Chicken and vegetable soup."

[0186] As described above, the cooking assistance system of the present invention can provide users with customized recipes and procedures, professional technical information, and healthy menus, thereby providing practical and effective support.

[0187] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0188] Step 1:

[0189] Collection of User Information

[0190] Input: The user enters their name, age, food preferences, allergy information, and health information into the terminal.

[0191] Processing: The terminal sends the entered information to the server, which receives the information and processes it to save it in a database. Specifically, the data sent from the terminal undergoes validation checks on the server side before being saved in a database (for example, MySQL or MongoDB).

[0192] Output: Save to database, and user information is stored on the server.

[0193] Specific operation: When the user presses the send button, the message "Sent. Thank you." is displayed on the terminal.

[0194] Step 2:

[0195] Providing cooking recipes

[0196] Input: The user types a request into the terminal, such as "Tell me some recipes using tomatoes."

[0197] Processing: The device sends this request to the server, which then sends the received request and the user's profile information to the generative AI model (e.g., OpenAI's GPT-3). The generative AI model generates the optimal recipe based on the prompt, which includes the user's profile information (such as name, age, and food preferences).

[0198] Output: The recipe information generated by the generative AI model is sent back to the server, which then sends it to the device and displays it to the user.

[0199] Specific operation: An image of "Tomato and Basil Pasta" and recipe instructions will be displayed on the device screen.

[0200] Step 3:

[0201] Real-time assistance with cooking procedures

[0202] Input: The user types a request into the terminal, such as "Tell me what to do next."

[0203] Processing: The device sends a request to the server, which then sends it to the generative AI model to generate the next step. The prompt contains the current step and the next step. The generative AI model generates the next step and sends the result back to the server.

[0204] Output: The generated next step information is sent from the server to the terminal and provided to the user by voice or text.

[0205] Specific operation: The device will say, "Please finely chop the tomatoes."

[0206] Step 4:

[0207] Providing professional technical information

[0208] Input: A user types a request into a device, such as "Tell me some pro tips for this recipe."

[0209] Processing: The device sends a request to the server, which queries a professional technical information database to obtain information and optimizes it for the generative AI model. The prompt contains the recipe and related technical information. The generative AI model returns the optimized technical information to the server.

[0210] Output: The optimized technical information is sent from the server to the terminal and displayed or explained to the user by voice.

[0211] Specific action: The information displayed is, "Professionals blanch tomatoes before chopping them to make them taste even better."

[0212] Step 5:

[0213] Healthy menu suggestions

[0214] Input: The user inputs a request into the terminal, such as "Tell me some healthy meals."

[0215] Processing: The device sends a request to the server, which then sends a prompt to the generative AI model to generate an optimal menu based on the user's health checkup data and nutritional balance information. The generative AI model generates a menu that takes nutritional balance into consideration and returns the results to the server.

[0216] Output: The generated menu information is sent from the server to the terminal and displayed to the user.

[0217] Specific behavior: The menu will be displayed: "Breakfast: Oatmeal and fruit, Lunch: Grilled salmon and salad, Dinner: Chicken and vegetable soup."

[0218] (Application example 1)

[0219] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0220] Currently, there are a wide variety of systems that provide cooking instructions and recipes, but these systems generally do not take into account the user's individual preferences, health status, or the specific ingredients used. Even in brick-and-mortar stores, customers are rarely offered recipe suggestions based on the ingredients they have on hand, and it is difficult to obtain professional technical advice or next cooking steps in real time. As a result, users often have to go through a lot of effort to obtain the information they want, resulting in a lack of convenience.

[0221] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0222] In this invention, the server includes a means for inputting user information and transmitting it to the server, a means for saving the input user information, and a means for causing a recipe generation AI to generate a recipe based on designated ingredients and providing the recipe to the user. This makes it possible to propose recipes based on ingredients in-store using advanced information terminals. It also makes it possible to provide the user with next steps in real time by voice or text.

[0223] Furthermore, by providing a means to acquire professional technical information, optimize it, and provide it to users, it is possible to convey professional cooking techniques and tips to users. In addition, by using health-related information to generate healthy menus using artificial intelligence, it is possible to provide these menus to users, making it possible to suggest optimal menus based on each individual's health status. This will greatly improve user convenience and provide optimal cooking support.

[0224] "Interactive generative artificial intelligence" is an artificial intelligence that collects information through dialogue with users and generates and provides appropriate information and services based on that information.

[0225] A "cooking assistance system" is a system that assists users when cooking by suggesting recipes and guiding them through cooking procedures.

[0226] "User Information" refers to personal information such as the user's name, age, food preferences, allergy information, and health information.

[0227] A "server" is a computer system used to store user information, process data, and generate recipes.

[0228] "Advanced information terminals" refer to electronic devices with advanced functions, such as smartphones and digital kiosks.

[0229] A "recipe" is a collection of information on how to make a dish, such as the ingredients to be used, their amounts, and cooking steps.

[0230] "Professional technical information" refers to information related to cooking techniques and tips possessed by professional chefs.

[0231] "Health-related information" refers to data related to maintaining and improving health, such as a user's health checkup data and nutritional balance information.

[0232] A "menu" refers to a meal combination or menu plan for a specific period of time.

[0233] The present invention relates to a cooking assistance system that uses interactive generative artificial intelligence and aims to provide optimal assistance to users when cooking. Specific embodiments will be described below.

[0234] 1. System Configuration

[0235] This system consists of a terminal for inputting user information, a server for storing and processing data, and multiple functions that utilize the generative AI model. The terminals are advanced information terminals such as smartphones and digital kiosks. The server stores user information and processes the data, and generates various data using a generative AI model (e.g., OpenAI's GPT-4).

[0236] 2. Collection of User Information

[0237] The server collects user information (such as name, age, food preferences, allergy information, and health information) sent from the device and stores it in a database, enabling it to provide customized services tailored to each individual user.

[0238] 3. Recipe Generation

[0239] When a user wants to know a recipe based on specific ingredients, the device sends the request to the server. The server uses the user information and the request to generate the optimal recipe using a generative AI model, which then sends the recipe back to the device. For example, a recipe such as "pasta with tomatoes and basil" is generated and served to the user.

[0240] 4. Providing real-time procedures

[0241] When a user wants real-time instructions for cooking a dish, the device sends a request to the server, which uses a generative AI model to generate the next steps and sends them back to the device in real time, providing specific instructions via voice or text, such as "finely chop the tomatoes."

[0242] 5. Providing professional technical information

[0243] When a user wants to know professional technical information for a particular recipe, the device sends the request to the server. The server queries a database of professional technical information, optimizes the information with a generative AI model, and provides it to the user. For example, the information provided may be, "Professionals recommend blanching the tomatoes before chopping them to enhance the flavor."

[0244] 6. Healthy meal suggestions

[0245] When a user wants to know about healthy meals, the device sends a request to the server. The server uses the user's health checkup data and nutritional balance information to generate an optimal meal plan using the generative AI model, and sends the plan back to the device. For example, a menu such as "Breakfast: Oatmeal and fruit, Lunch: Grilled salmon and salad, Dinner: Chicken and vegetable soup" may be provided.

[0246] Hardware and Software

[0247] The server is built using programming languages ​​and frameworks such as Python and Flask, and uses OpenAI's API for generative AI, with user information stored in a data storage system such as an SQLite database.

[0248] Specific examples

[0249] Examples of prompts include:

[0250] "Tell me a recipe that uses tomatoes, basil, and pasta."

[0251] "Tell me the next cooking step"

[0252] "What are some pro tips for this recipe?"

[0253] By using these prompts, users can obtain optimal recipes, cooking procedures, and specialized technical information in real time. In this way, a cooking assistance system using interactive generative AI can provide users with highly convenient and effective support.

[0254] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0255] Step 1:

[0256] Collection of User Information

[0257] Users enter their name, age, food preferences, allergy information, and health information through a device (smartphone or digital kiosk). The entered data is sent from the device to a server, which receives the data and stores it in an SQLite database. This creates the foundation for providing services customized for each user.

[0258] Step 2:

[0259] Recipe Generation

[0260] When a user wants to know a recipe based on a specific ingredient, they can input, for example, "Tell me a recipe using tomatoes" into their device. The device then sends this request to the server. The server references the user information and the request and generates the optimal recipe based on a generative AI model (for example, GPT-4). The generated recipe is then sent back from the server to the device and displayed to the user. Specifically, in response to the prompt "Tell me a recipe using tomatoes, basil, and pasta," "Pasta with tomatoes and basil" is suggested.

[0261] Step 3:

[0262] Providing real-time procedures

[0263] When a user wants real-time support for cooking steps, they can type, for example, "Tell me the next steps" into their device. Upon receiving this request, the device sends a request to the server. The server generates the next cooking steps based on a generative AI model. These steps are sent back from the server to the device and provided to the user via text or voice. Specifically, instructions such as "Please finely chop the tomatoes" are provided in real time.

[0264] Step 4:

[0265] Providing professional technical information

[0266] If a user wants to know professional technical information for a particular recipe, they can type, for example, "Tell me the pro tips for this recipe" into their device. The device then sends this request to the server. The server queries a database of professional technical information and optimizes the information using a generative AI model. The optimized information is then sent back from the server to the device and presented to the user via display or audio. For example, they might provide information such as, "Professionals blanch tomatoes before chopping them to enhance their flavor."

[0267] Step 5:

[0268] Healthy menu suggestions

[0269] If a user wants to know what a healthy menu is, they can type, for example, "Tell me what a healthy menu is." The device sends this request to the server. The server uses the user's health checkup data and nutritional balance information to have the generative AI model generate the optimal menu. This menu information is sent back from the server to the device and provided to the user. Specifically, the suggested menu might be "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup."

[0270] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0271] The present invention is a cooking assistance system that utilizes an emotion engine in addition to interactive generative artificial intelligence, and provides more personalized assistance by recognizing the user's emotions and customizing recipes and procedures based on those emotions, and adjusting the tone and content of the dialogue. Specific embodiments of the present invention are described below.

[0272] System Overview

[0273] Users can interact with the system through an interface and receive various cooking assistance. The system utilizes information from the server, the device, and the user, and also uses an emotion engine to suggest recipes, provide real-time assistance with procedures, provide professional technical information, and suggest healthy meals.

[0274] Program processing overview

[0275] The system works through user information collection, recipe generation, real-time assistance, professional technical information provision, healthy meal suggestions, as well as emotion recognition and optimization using an emotion engine.

[0276] Collection of User Information

[0277] 1. The user launches the app for the first time and is prompted to enter user information.

[0278] 2. The device displays fields for the user to enter their name, age, food preferences, allergy information, and health information.

[0279] 3. The user enters this information and presses the submit button.

[0280] 4. The terminal sends the entered information to the server.

[0281] 5. The server stores the received user information in a database.

[0282] Providing cooking recipes

[0283] 1. A user voices or texts a request for a recipe using a specific ingredient (e.g., "Tell me some recipes using tomatoes").

[0284] 2. The terminal sends the input request to the server.

[0285] 3. The server uses the AI ​​to generate the optimal recipe based on the user's profile information and request.

[0286] 4. The generation AI returns the generated recipe information to the server.

[0287] 5. The server sends the recipe information to the terminal, and the terminal displays the generated recipe (e.g., "Pasta with Tomato and Basil") for the user.

[0288] Real-time assistance with cooking procedures

[0289] 1. If a user is cooking and wants to know the next step, they can send a request via voice or text (e.g., "Tell me the next step").

[0290] 2. The device sends a request to the server.

[0291] 3. The server has the generation AI generate the next steps and send those steps back to the server.

[0292] 4. The generative AI generates next step information (e.g., "Please finely chop the tomatoes").

[0293] 5. The server receives the generated procedure information and sends it to the terminal.

[0294] 6. The device will provide the user with next steps via voice or text.

[0295] Providing professional technical information

[0296] 1. A user submits a request for pro technical information on a recipe (e.g., "What are some pro tips for this recipe?").

[0297] 2. The device sends a request to the server.

[0298] 3. The server queries a professional technical information database and provides that information to the generation AI.

[0299] 4. Generative AI optimizes technical information and generates specific advice (e.g., "Professionals recommend blanching tomatoes before chopping them to make them taste even better").

[0300] 5. The server receives the optimized technical information and sends it to the device.

[0301] 6. The device displays or audibly explains technical information to the user.

[0302] Healthy menu suggestions

[0303] 1. If a user wants to know about healthy meals, they send a request (e.g., "Tell me about healthy meals").

[0304] 2. The device sends a request to the server.

[0305] 3. The server retrieves the user's health checkup data and nutritional balance information from the database and provides it to the generation AI.

[0306] 4. Generative AI generates nutritionally balanced menus based on health information (e.g., "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup").

[0307] 5. The server receives the generated menu information and sends it to the terminal.

[0308] 6. The device displays the menu information to the user.

[0309] Utilizing the Emotion Engine

[0310] 1. If the user's emotions change during a conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion.

[0311] 2. The device sends the recognized emotion data to the server.

[0312] 3. The server provides the generative AI with emotional data and issues instructions to customize the recipe, steps, and dialogue content.

[0313] For example, if a user is feeling stressed about cooking, the emotion engine will recognize this and the generative AI will suggest recipes that are easy and quick to make. This allows users to receive optimal cooking assistance tailored to their mood and situation.

[0314] In this way, a cooking assistance system that combines interactive generative artificial intelligence and an emotion engine can provide users with customized recipes and procedures, professional techniques, and healthy menus, and can also respond to their emotions, providing more personalized assistance.

[0315] The processing flow will be explained below.

[0316] Processing flow of a cooking support system using an emotion engine

[0317] Collection of User Information

[0318] Step 1:

[0319] The user launches the app for the first time and is presented with a screen to enter user information.

[0320] Step 2:

[0321] The device will present the user with fields to enter their name, age, food preferences, allergy information, and health information.

[0322] Step 3:

[0323] The user enters this information and presses the send button.

[0324] Step 4:

[0325] The terminal transmits the input information to the server.

[0326] Step 5:

[0327] The server stores the received user information in a database.

[0328] Providing cooking recipes

[0329] Step 1:

[0330] The user voices or types in a request for a recipe using a specific ingredient (e.g., "Tell me some recipes using tomatoes").

[0331] Step 2:

[0332] The terminal transmits the input request to the server.

[0333] Step 3:

[0334] The server references the user's profile information from the database and sends the request to the generation artificial intelligence.

[0335] Step 4:

[0336] Generative AI generates the optimal recipe based on the user's profile and specified ingredients.

[0337] Step 5:

[0338] The server receives the generated recipe information and transmits it to the terminal.

[0339] Step 6:

[0340] The device displays the generated recipe (e.g., "Pasta with Tomato and Basil") to the user.

[0341] Real-time assistance with cooking procedures

[0342] Step 1:

[0343] If a user is cooking and wants to know the next step, they can send a request by voice or text (e.g., "Tell me the next step").

[0344] Step 2:

[0345] The device sends a request to the server.

[0346] Step 3:

[0347] The server issues instructions to the generation artificial intelligence to generate the next step.

[0348] Step 4:

[0349] Generative AI generates next step information (e.g., "Please finely chop the tomatoes").

[0350] Step 5:

[0351] The server receives the generated procedure information and transmits it to the terminal.

[0352] Step 6:

[0353] The device will provide the user with next steps via voice or text.

[0354] Providing professional technical information

[0355] Step 1:

[0356] A user submits a request for pro technical information on a recipe (e.g., "What are some pro tips for this recipe?").

[0357] Step 2:

[0358] The device sends a request to the server.

[0359] Step 3:

[0360] The server queries a professional technical information database and provides the information to the generating artificial intelligence.

[0361] Step 4:

[0362] Generative AI optimizes technical information and generates specific advice (e.g., "Professionals recommend blanching tomatoes before chopping them to make them taste even better").

[0363] Step 5:

[0364] The server receives the optimized technical information and sends it to the terminal.

[0365] Step 6:

[0366] The terminal displays or audibly explains technical information to the user.

[0367] Healthy menu suggestions

[0368] Step 1:

[0369] If a user wants to know about healthy meals, they send a request (e.g., "Tell me about healthy meals").

[0370] Step 2:

[0371] The device sends a request to the server.

[0372] Step 3:

[0373] The server retrieves the user's health checkup data and nutritional balance information from the database and provides it to the generating artificial intelligence.

[0374] Step 4:

[0375] Generative artificial intelligence generates nutritionally balanced menus based on health information (e.g., "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup").

[0376] Step 5:

[0377] The server receives the generated menu information and transmits it to the terminal.

[0378] Step 6:

[0379] The terminal displays the menu information to the user.

[0380] Utilizing the Emotion Engine

[0381] Step 1:

[0382] If the user's emotions change during the conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion.

[0383] Step 2:

[0384] The device transmits the recognized emotion data to the server.

[0385] Step 3:

[0386] The server provides emotional data to the generative AI, which then issues instructions to customize recipes, steps, and dialogue content.

[0387] As a concrete example, if a user is feeling stressed about cooking, the emotion engine will recognize this and the generative AI will suggest recipes that are easy and quick to make. In this example, it is possible to suggest "time-saving recipes that can be made in the microwave." This allows users to receive optimal cooking support tailored to their mood and situation.

[0388] Example 2

[0389] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0390] In today's modern lifestyles, it is extremely important to provide healthy and efficient cooking assistance that accommodates the different lifestyles and food preferences of each individual user. However, conventional cooking assistance systems lack the ability to customize based on the user's emotions and health status, making it difficult to provide more personalized assistance. There is also a need to effectively incorporate technical information from experts to improve the quality of cooking. Furthermore, in situations where real-time response is required, providing prompt and appropriate information has been insufficient.

[0391] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0392] In this invention, the server includes means for inputting user information and transmitting it to the server, means for saving the input user information, means for having a generation AI generate a recipe based on specified ingredients and providing the recipe to the user, means for providing the user with real-time audio or text instructions for the next steps, means for acquiring technical information from experts and optimizing it to provide to the user, means for having the generation AI generate a healthy menu based on health-related information and providing the menu to the user, and means for analyzing the user's emotions using an emotion recognition engine and customizing the recipe, steps, and dialogue content based on the emotions. This enables personalized cooking support according to the individual needs and emotions of the user, provision of menus that accommodate health management, and high-quality cooking support utilizing the skills of experts.

[0393] "User information" refers to the name, age, food preferences, allergy information, health information, etc. entered by the user.

[0394] "Server" refers to a computer system that stores input user information, processes various data, and provides it to the generation AI.

[0395] "Specified ingredients" refers to ingredients that the user specifies to be used in creating the recipe.

[0396] "Generative AI" refers to an AI model that generates recipes, steps, menus, etc. based on user requests.

[0397] "Real-time" refers to nearly immediate response to user requests.

[0398] "Expert technical information" refers to information about professional knowledge and techniques related to cooking.

[0399] "Health-related information" refers to all data related to health and nutrition, such as a user's health checkup data and nutritional balance information.

[0400] An "emotion recognition engine" refers to a system that recognizes emotions by analyzing the user's voice and facial expressions.

[0401] "Personalized cooking assistance" refers to providing cooking assistance customized based on the user's individual information and emotions.

[0402] "Customizing dialogue content" refers to adjusting the tone and content of dialogue provided according to the user's emotions and situation.

[0403] The present invention is a cooking assistance system that utilizes interactive generative artificial intelligence (generative AI model) and an emotion engine to recognize the user's emotions and customize recipes and procedures according to those emotions, thereby providing more personalized assistance. A specific embodiment of this system is shown below.

[0404] Overall system overview

[0405] Users can interact with the system through devices such as smartphones and tablets and receive various cooking assistance. The system utilizes information from the server, devices, and users, and uses an emotion engine to suggest recipes, provide real-time assistance with procedures, provide expert technical information, and suggest healthy meals. The system mainly uses the following hardware and software:

[0406] Hardware: smartphones, tablets, servers, database servers

[0407] Software: Interactive generative AI models, emotion recognition engines, user interface applications, database management systems

[0408] Program processing overview

[0409] The system works through user information collection, recipe generation, real-time assistance, expert technical input, healthy meal suggestions, as well as emotion recognition and optimization using an emotion engine.

[0410] Collection of User Information

[0411] When a user launches the app for the first time, a screen for entering user information is displayed. The device displays fields for entering the user's name, age, food preferences, allergy information, and health information. When the user enters this information and presses the send button, the device sends the entered information to the server. The server stores the received user information in a database.

[0412] Providing cooking recipes

[0413] A user inputs a request for a recipe using a specific ingredient via voice or text. For example, if the request is "Tell me a recipe using tomatoes," the device analyzes the input request and sends it to the server. The server uses the user's profile information and the request content to have the generative AI model generate the optimal recipe. The generative AI generates recipe information, such as "pasta with tomatoes and basil," and sends it back to the server. The server then sends the generated recipe information to the device and displays it to the user.

[0414] Real-time assistance with cooking procedures

[0415] When a user wants to know the next step while cooking, they send a request by voice or text. For example, if a request is made to "tell me the next step," the device sends the request to the server. The server then has the generative AI model generate the next step and sends that step back to the server. The generative AI generates step information such as "finely chop the tomatoes," and the server sends that information to the device. The device then provides the next step to the user by voice or text.

[0416] Providing technical information from experts

[0417] A user sends a request for expert technical information about a recipe. For example, if the user requests, "Tell me the pro tips for this recipe," the device sends the request to the server. The server queries a database of expert technical information and provides that information to the generative AI model. The generative AI generates specific technical information, such as "Professionals recommend blanching tomatoes before chopping them to enhance their flavor," and sends it back to the server. The server then sends that information to the device and explains it to the user by displaying it or by voice.

[0418] Healthy menu suggestions

[0419] When a user wants to know about healthy menu options, they send a request. For example, if the request is "Tell me about healthy menu options," the device sends the request to the server. The server retrieves the user's health checkup data and nutritional balance information from a database and provides this to the generative AI model. The generative AI generates a healthy menu option such as "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup." The server sends the generated menu information to the device and displays it to the user.

[0420] Utilizing the Emotion Engine

[0421] If the user's emotions change during a conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion. The device then sends the recognized emotion data to the server, which then provides the emotion data to the generative AI model and issues instructions to customize the recipe, steps, and conversation content. For example, if the user is feeling stressed about cooking, the emotion engine will recognize this, and the generative AI model will suggest a recipe that is easy and quick to make. This allows the user to receive optimal cooking assistance tailored to their mood and situation.

[0422] Examples of prompt statements

[0423] Recipe generation prompt: "Generate a quick recipe using tomatoes. The user is stressed."

[0424] Technical information prompt: "What are some pro tips for this recipe?"

[0425] In this way, the cooking assistance system of the present invention combines interactive generative artificial intelligence with an emotion engine to provide users with customized recipes, procedures, expert techniques, and healthy menus, and can also respond to emotions, making it possible to provide more personalized assistance.

[0426] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0427] Collection of User Information

[0428] Step 1:

[0429] The user launches the app for the first time and is presented with a screen to enter user information.

[0430] What it does: The device displays fields for entering name, age, food preferences, allergy information, and health information.

[0431] Step 2:

[0432] The user enters this information and presses the send button.

[0433] Specific operation: The device receives the entered information, detects when the send button is pressed, and sends it to the server.

[0434] Input: User's personal information (name, age, food preferences, allergy information, health information)

[0435] Output: User information sent to the server

[0436] Step 3:

[0437] The terminal transmits the input information to the server.

[0438] Specific operation: The terminal sends the received user information to the server via a secure communication channel.

[0439] Input: User information (secure format)

[0440] Output: User information sent to the server

[0441] Step 4:

[0442] The server stores the received user information in a database.

[0443] Specific operation: The server stores the received user information in a database and makes it available for subsequent processes.

[0444] Input: User information sent to the server

[0445] Output: User information stored in the database

[0446] Providing cooking recipes

[0447] Step 1:

[0448] A user voices or texts a request for a recipe using a specific ingredient.

[0449] Specific action: For example, request "Tell me some recipes that use tomatoes."

[0450] Input: Request for a specific material

[0451] Output: The text or audio data of the request

[0452] Step 2:

[0453] The terminal transmits the input request to the server.

[0454] Specific operation: The device performs voice recognition and text analysis and sends the request content to the server.

[0455] Input: The text or audio data of the request

[0456] Output: The request sent to the server

[0457] Step 3:

[0458] The server generates a prompt for the generative AI model based on the user's profile information and request content.

[0459] Specific operation: The server converts the user information and request content into a prompt text.

[0460] Input: User profile information and request details

[0461] Output: Prompt sentence for generative AI model

[0462] Step 4:

[0463] The generative AI model generates recipe information based on the prompt sentence and sends it back to the server.

[0464] Specific behavior: The generative AI generates specific recipes such as "pasta with tomato and basil."

[0465] Input: prompt statement

[0466] Output: Generated recipe information

[0467] Step 5:

[0468] The server transmits the generated recipe information to the terminal.

[0469] Specific operation: The server transfers the recipe information to the device.

[0470] Input: Generated recipe information

[0471] Output: Recipe information sent to the device

[0472] Step 6:

[0473] The terminal displays the generated recipe to the user.

[0474] Specific operation: The device displays the received recipe information on the user interface.

[0475] Input: Recipe information sent to the device

[0476] Output: Recipe information displayed to the user

[0477] Real-time assistance with cooking procedures

[0478] Step 1:

[0479] If a user is cooking and wants to know the next step, they can send a request by voice or text.

[0480] Specific behavior: For example, request "Tell me what the next step is."

[0481] Input: Next Step Request

[0482] Output: The text or audio data of the request

[0483] Step 2:

[0484] The device sends a request to the server.

[0485] Specific operation: The device performs voice recognition or text analysis and forwards the request content to the server.

[0486] Input: The text or audio data of the request

[0487] Output: The request sent to the server

[0488] Step 3:

[0489] The server asks the generative AI model to generate the next step as a prompt.

[0490] Specific operation: The server summarizes the current procedure information into a prompt sentence and passes it to the generative AI model.

[0491] Input: Current procedure information

[0492] Output: Prompt sentence for generative AI model

[0493] Step 4:

[0494] The generative AI model generates next step information and sends it back to the server.

[0495] Specific actions: The generative AI generates specific steps such as "finely chop the tomatoes."

[0496] Input: prompt statement

[0497] Output: Generated procedure information

[0498] Step 5:

[0499] The server transmits the generated procedure information to the terminal.

[0500] Specific operation: The server transfers the generated procedure information to the terminal.

[0501] Input: Generated procedure information

[0502] Output: Instructions sent to the terminal

[0503] Step 6:

[0504] The device will provide the user with next steps via voice or text.

[0505] Specific operation: The device explains the received procedure information to the user by voice or text.

[0506] Input: Instructions sent to the terminal

[0507] Output: Next steps information provided to the user

[0508] Providing technical information from experts

[0509] Step 1:

[0510] A user submits a request for expert technical information on a recipe.

[0511] Specific behavior: For example, request "Tell me some pro tips for this recipe."

[0512] Input: Technical Information Request

[0513] Output: The text or audio data of the request

[0514] Step 2:

[0515] The device sends a request to the server.

[0516] Specific operation: The device performs voice recognition or text analysis and forwards the request content to the server.

[0517] Input: The text or audio data of the request

[0518] Output: The request sent to the server

[0519] Step 3:

[0520] The server queries a database of technical information from experts.

[0521] Specific operation: The server refers to the database based on the request and retrieves technical information.

[0522] Input: Request details

[0523] Output: Expert technical information

[0524] Step 4:

[0525] The generative AI model generates specific, optimized advice based on technical information and sends it back to the server.

[0526] Specific operation: The generative AI generates advice such as, "Professionals recommend blanching tomatoes before chopping them to make them taste even better."

[0527] Input: Expert technical information

[0528] Output: The generated advice

[0529] Step 5:

[0530] The server transmits the generated advice to the terminal.

[0531] Specific operation: The server transfers the generated advice to the terminal.

[0532] Input: Generated advice

[0533] Output: Advice sent to terminal

[0534] Step 6:

[0535] The terminal displays or audibly explains the advice to the user.

[0536] Specific operation: The device displays the received advice on the user interface or explains it aloud.

[0537] Input: Advice sent to terminal

[0538] Output: Advice given to the user

[0539] Healthy menu suggestions

[0540] Step 1:

[0541] If a user wants to know about healthy meals, they send a request.

[0542] Specific action: For example, request "Tell me some healthy meals."

[0543] Input: Healthy Meal Request

[0544] Output: The text or audio data of the request

[0545] Step 2:

[0546] The device sends a request to the server.

[0547] Specific operation: The device performs voice recognition or text analysis and forwards the request content to the server.

[0548] Input: The text or audio data of the request

[0549] Output: The request sent to the server

[0550] Step 3:

[0551] The server retrieves the user's health checkup data and nutritional balance information from the database.

[0552] Specific operation: The server retrieves data related to the user's health from the database.

[0553] Input: User's health checkup data, nutritional balance information

[0554] Output: Health data collected by the server

[0555] Step 4:

[0556] The generative AI model generates menus that take nutritional balance into consideration based on health information.

[0557] Specific operation: The generative AI generates menus such as "Breakfast: oatmeal and fruit, Lunch: grilled salmon and salad, Dinner: chicken and vegetable soup."

[0558] Input: User's health checkup data and nutritional balance information

[0559] Output: Generated healthy meal plan

[0560] Step 5:

[0561] The server transmits the generated menu information to the terminal.

[0562] Specific operation: The server transfers the generated menu information to the terminal.

[0563] Input: Generated menu information

[0564] Output: Menu information sent to the device

[0565] Step 6:

[0566] The terminal displays the menu information to the user.

[0567] Specific operation: The device displays the received menu information on the user interface.

[0568] Input: Menu information sent to the device

[0569] Output: Menu information provided to the user

[0570] Utilizing the Emotion Engine

[0571] Step 1:

[0572] If the user's emotions change during the conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion.

[0573] How it works: The emotion engine analyzes the user's voice and facial expressions in real time to detect emotions such as stress or joy.

[0574] Input: User voice and facial expression data

[0575] Output: Recognized emotion data

[0576] Step 2:

[0577] The device transmits the recognized emotion data to the server.

[0578] Specific operation: The device sends emotion data to the server.

[0579] Input: Recognized emotion data

[0580] Output: Emotion data sent to the server

[0581] Step 3:

[0582] The server provides emotional data to a generative AI model, which then provides instructions to customize recipes, steps, and dialogue.

[0583] Specific operation: The server incorporates emotional data into a prompt sentence and passes it to the generative AI model.

[0584] Input: Emotion data

[0585] Output: Prompt sentence for generative AI model

[0586] Step 4:

[0587] The generative AI model generates emotion-based recipes and instructions and sends them back to the server.

[0588] For example, if a user is feeling stressed about cooking, the generative AI will generate a recipe such as, "If you're feeling stressed, make tomato and basil pasta. It's easy to make."

[0589] Input: prompt statement

[0590] Output: The generated recipe or instructions

[0591] Step 5:

[0592] The server sends the generated recipes and instructions to the device.

[0593] Specific operation: The server transfers the generated information to the terminal.

[0594] Input: Generated recipes and instructions

[0595] Output: Information sent to the terminal

[0596] Step 6:

[0597] The device displays or audibly explains the recipe and steps to the user.

[0598] Specific operation: The device displays or audibly explains the received information.

[0599] Input: Information sent to the device

[0600] Output: The recipe or instructions provided to the user

[0601] (Application example 2)

[0602] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0603] Conventional cooking assistance systems and production process management systems did not provide support that took into account the emotions of users and workers. As a result, they were unable to respond appropriately to situations that caused stress to users and workers, leading to problems such as reduced work efficiency and decreased motivation. In addition, it was difficult to respond to the individual needs of users and workers, resulting in a lack of personalized support.

[0604] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting user information and transmitting it to the server, means for saving the input user information, means for having a generation AI generate a recipe based on specified ingredients and providing the recipe to the user, means for providing the user with real-time audio or text instructions on how to proceed, means for acquiring professional technical information, optimizing it, and providing it to the user, means for collecting emotion data using an emotion engine that recognizes the user's emotions and customizing the content of the provision based on that emotion data, means for having a generation AI generate optimal support information based on the emotion data and work content of factory workers and providing it to the workers, and means for having a generation AI generate healthy menus based on health-related information and providing the menus to the user. This provides personalized support that takes the emotions of users and workers into consideration, thereby reducing stress and improving work efficiency.

[0605] "Interactive generative artificial intelligence" is an artificial intelligence that collects information through dialogue with the user and makes appropriate suggestions and assistance based on that information.

[0606] A "system" is a collection of devices and software in which multiple elements work together to achieve a specific function.

[0607] "User Information" means certain data about you, such as your name, age, food preferences, allergy information, and health information.

[0608] A "server" is a computer system that provides services and data to other computers on a network.

[0609] An "emotion engine" is software that analyzes the user's voice and facial expressions to recognize emotions and process information accordingly.

[0610] A "recipe" is a list of steps and ingredients for making a particular dish.

[0611] "Professional technical information" refers to specific techniques and know-how provided by experts.

[0612] A "healthy menu" is a combination of nutritionally balanced meals.

[0613] A "factory work support system" is a system that aims to improve the efficiency of work within a factory and increase quality and productivity.

[0614] "Workers" are workers who work in factories and other places.

[0615] "Emotion data" refers to information about the emotions of users and workers obtained by the emotion engine.

[0616] This invention is a factory work support system that uses interactive generative artificial intelligence and an emotion engine, and provides personalized support according to the emotions of users and workers.

[0617] System Overview

[0618] This system includes a terminal, a server, an emotion engine, and a generative AI model. The terminal provides an interface for users and workers to input information, and the server stores and processes the input information and generates appropriate support information.

[0619] Hardware and Software

[0620] 1. Hardware:

[0621] Cameras for facial recognition (e.g., high-performance webcams)

[0622] Microphone for voice analysis (e.g. high-sensitivity microphone)

[0623] Terminals for workers (e.g., tablet terminals)

[0624] 2. Software:

[0625] Emotion recognition libraries (e.g., emotion analysis APIs)

[0626] Generative AI models (e.g., interactive generative AI engines)

[0627] Database (e.g. SQL database)

[0628] Front-end (e.g., web browser-based interface)

[0629] Data processing and calculation

[0630] The server receives the user information and worker information sent from the terminal and performs the following data processing and calculations.

[0631] 1. Emotion recognition:

[0632] Cameras and microphones are used to collect the facial expressions and voices of workers.

[0633] Using the sentiment analysis API, the collected data is analyzed to obtain sentiment data.

[0634] 2. Generating support information:

[0635] Based on emotion data and task data, the generative AI model generates appropriate work procedures and support information.

[0636] The generated information is stored in a database and transmitted to the terminal as needed.

[0637] Specific use cases

[0638] Let's say a worker is performing assembly work in a factory. While working, a camera and microphone analyze the worker's facial expressions and voice, and an emotion engine detects stress. The server receives the stress data and sends a prompt to the generation AI saying, "The worker is in a stressful state, so please suggest a simplified procedure for the work." The generation AI generates a simplified procedure, and the server sends that information to the terminal, presenting the simple procedure to the worker.

[0639] Prompt Sentence Examples

[0640] "High Stress" state. Please simplify the current task and suggest next steps.

[0641] In this way, a factory work support system that combines interactive generative artificial intelligence and an emotion engine can provide flexible support that responds to the emotions of workers, improving work efficiency and worker satisfaction.

[0642] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0643] Step 1:

[0644] The terminal provides an interface for inputting user information. The user or worker inputs user information such as name, age, food preferences, allergy information, and health information, and presses the send button. The input data is sent to the server.

[0645] Step 2:

[0646] The server stores the received user information in a database, including user and worker profile information.

[0647] Step 3:

[0648] A request for cooking or task assistance is entered via voice or text from the device. For example, a request might be "Tell me a recipe using tomatoes" or "Tell me the next step in the task." The request is then sent to the server.

[0649] Step 4:

[0650] The server sends a prompt to the generative AI model based on the request content and user information to generate the optimal recipe and work procedure. An example of a prompt sentence is, "The worker is under stress, so please suggest a procedure that simplifies the work."

[0651] Step 5:

[0652] A generative AI model generates recipes and instructions based on the prompt, which are then sent back to the server. Examples include "pasta with tomatoes and basil" and instructions such as "finely chop the tomatoes."

[0653] Step 6:

[0654] The server sends the information received from the generation AI to the terminal, which then displays or speaks the generated recipes and work procedures to the user or worker.

[0655] Step 7:

[0656] The device's camera and microphone analyze the facial expressions and voices of users and workers in real time. The emotion engine recognizes emotions based on the analyzed data and sends specific emotional data to the server.

[0657] Step 8:

[0658] Based on the emotion data received by the server, the generative AI model is prompted again to generate assistance according to the emotion, resulting in the generation of customized recipes and work procedures according to the emotion.

[0659] Step 9:

[0660] The server sends the regenerated information to the terminal, which then presents the user or worker with optimized recipes and work procedures, reducing stress and enabling more efficient work.

[0661] The above processing steps realize a factory work support system that utilizes interactive generative AI and an emotion engine. The collaboration between the generative AI model and the emotion engine makes it possible to provide personalized support according to the emotions of users and workers.

[0662] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0663] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0664] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0665] [Second embodiment]

[0666] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0667] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0668] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0669] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0670] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0671] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0672] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0673] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0674] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0675] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0676] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0677] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0678] The present invention provides a cooking assistance system that utilizes interactive generative artificial intelligence. Specific embodiments of the system will be described below.

[0679] System Overview

[0680] Users can interact with the system through an interface and receive various cooking assistance. The system primarily utilizes information collected from the server, terminals, and users to suggest recipes, provide real-time assistance with procedures, provide professional technical information, and suggest health-based menus.

[0681] Program processing overview

[0682] The system operates through the following steps: collecting user information, generating recipes, providing real-time assistance, providing professional technical information, and suggesting healthy meals.

[0683] Collection of User Information

[0684] 1. When a user uses the system for the first time, the terminal displays a screen for the user to enter their name, age, food preferences, allergy information, and health information.

[0685] 2. The user enters this information and presses the submit button.

[0686] 3. The device sends the entered information to the server.

[0687] 4. The server stores the received user information in a database.

[0688] 5. This allows the system to build a foundation for providing customized assistance to each user.

[0689] Providing cooking recipes

[0690] 1. If a user wants to know recipes that use a specific ingredient, they can request, for example, "Tell me recipes that use tomatoes."

[0691] 2. The device receives this request and sends it to the server.

[0692] 3. The server uses the AI ​​to generate the optimal recipe based on the user's profile information and request.

[0693] 4. The generation AI returns the generated recipe information to the server.

[0694] 5. The server sends the recipe information to the terminal, and the terminal displays the recipe to the user.

[0695] As a specific example, a recipe such as "tomato and basil pasta" is generated and provided to the user.

[0696] Real-time assistance with cooking procedures

[0697] 1. When the user actually starts cooking and wants to know the next steps, they can request, for example, "Tell me the next steps."

[0698] 2. The device sends this request to the server.

[0699] 3. The server has the generation AI generate the next steps and send those steps back to the server.

[0700] 4. The server sends the procedure information to the terminal, and the terminal provides the procedure to the user by voice or text.

[0701] For example, instructions such as "Please finely chop the tomatoes" are provided in real time.

[0702] Providing professional technical information

[0703] 1. If a user wants to know professional technical information about a particular recipe, they can request, for example, "Tell me the pro tips for this recipe."

[0704] 2. The device sends this request to the server.

[0705] 3. The server queries a professional technical information database and optimizes the information for the generating AI.

[0706] 4. The generating AI provides optimized technical information to the server.

[0707] 5. The server sends the technical information to the terminal, and the terminal displays or audibly explains the technical information to the user.

[0708] As a specific example, information such as "Professionals blanch tomatoes before chopping them to make them taste even better" is provided.

[0709] Healthy menu suggestions

[0710] 1. If a user wants to know about healthy meals, they can request, for example, "Tell me about healthy meals."

[0711] 2. The device sends this request to the server.

[0712] 3. The server uses the AI ​​to generate the optimal menu based on the user's health checkup data and nutritional balance information.

[0713] 4. The generation AI generates a menu that takes nutritional balance into consideration and sends it back to the server.

[0714] 5. The server sends the menu information to the terminal, and the terminal displays the menu to the user.

[0715] For example, a menu might include "Breakfast: oatmeal and fruit, Lunch: grilled salmon and salad, Dinner: chicken and vegetable soup."

[0716] In this way, a cooking assistance system utilizing interactive generative artificial intelligence can provide users with customized recipes and procedures, professional techniques, and healthy menus, providing practical and effective support.

[0717] The processing flow will be explained below.

[0718] Collection of User Information

[0719] Step 1:

[0720] The user launches the app for the first time and is presented with a screen to enter user information.

[0721] Step 2:

[0722] The device will present the user with fields to enter their name, age, food preferences, allergy information, and health information.

[0723] Step 3:

[0724] The user enters the required information and presses the send button.

[0725] Step 4:

[0726] The terminal transmits the input information to the server.

[0727] Step 5:

[0728] The server stores the received user information in a database.

[0729] Providing cooking recipes

[0730] Step 1:

[0731] The user voices or types in a request for a recipe using a specific ingredient (e.g., "Tell me some recipes using tomatoes").

[0732] Step 2:

[0733] The terminal transmits the input request to the server.

[0734] Step 3:

[0735] The server references the user's profile information from the database and sends the request to the generation artificial intelligence.

[0736] Step 4:

[0737] Generative AI generates the optimal recipe based on the user's profile and specified ingredients.

[0738] Step 5:

[0739] The server receives the generated recipe information and transmits it to the terminal.

[0740] Step 6:

[0741] The device displays the generated recipe (e.g., "Pasta with Tomato and Basil") to the user.

[0742] Real-time assistance with cooking procedures

[0743] Step 1:

[0744] If a user is cooking and wants to know the next step, they can send a request by voice or text (e.g., "Tell me the next step").

[0745] Step 2:

[0746] The device sends a request to the server.

[0747] Step 3:

[0748] The server issues instructions to the generation artificial intelligence to generate the next step.

[0749] Step 4:

[0750] Generative AI generates next step information (e.g., "Please finely chop the tomatoes").

[0751] Step 5:

[0752] The server receives the generated procedure information and transmits it to the terminal.

[0753] Step 6:

[0754] The device will provide the user with next steps via voice or text.

[0755] Step 7:

[0756] The user follows the instructions and sends the request again if the next step is required.

[0757] Providing professional technical information

[0758] Step 1:

[0759] A user submits a request for pro technical information on a recipe (e.g., "What are some pro tips for this recipe?").

[0760] Step 2:

[0761] The device sends a request to the server.

[0762] Step 3:

[0763] The server queries a professional technical information database and provides the information to the generating artificial intelligence.

[0764] Step 4:

[0765] Generative AI optimizes technical information and generates specific advice (e.g., "Professionals recommend blanching tomatoes before chopping them to make them taste even better").

[0766] Step 5:

[0767] The server receives the optimized technical information and sends it to the terminal.

[0768] Step 6:

[0769] The terminal displays or audibly explains technical information to the user.

[0770] Healthy menu suggestions

[0771] Step 1:

[0772] If a user wants to know about healthy meals, they send a request (e.g., "Tell me about healthy meals").

[0773] Step 2:

[0774] The device sends a request to the server.

[0775] Step 3:

[0776] The server retrieves the user's health information and nutritional balance information from the database and provides it to the generating artificial intelligence.

[0777] Step 4:

[0778] Generative artificial intelligence generates nutritionally balanced menus based on health information (e.g., "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup").

[0779] Step 5:

[0780] The server receives the generated menu information and transmits it to the terminal.

[0781] Step 6:

[0782] The terminal displays the menu information to the user.

[0783] The above are the specific processing steps of the cooking assistance system that uses interactive generative artificial intelligence.

[0784] Example 1

[0785] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0786] Current cooking assistance systems often fail to adequately address the individual needs and circumstances of users. Specifically, they lack customized recipes and assistance that reflect the user's health status, food preferences, allergy information, etc. They also lack real-time assistance needed during cooking and a means to easily obtain professional technical information. Furthermore, there is a lack of healthy menu suggestions based on the user's health information, so more personalized assistance is needed.

[0787] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0788] In this invention, the server includes means for inputting user information and transmitting it to the server, means for saving the input user information, means for having the AI ​​generate a recipe based on specified ingredients and providing the recipe to the user, means for providing the user with the next steps in real time by voice or text when the user actually starts cooking and requests the next steps, means for obtaining professional technical information for a specific recipe, optimizing it, and providing it to the user, and means for having the AI ​​generate a healthy menu based on health-related information and providing the menu to the user. This makes it possible to provide customized cooking support and real-time support that meets the individual needs of the user, as well as professional technical information and healthy menu suggestions.

[0789] "User information" is a general term for data related to individual needs and circumstances, such as a user's name, age, food preferences, allergy information, and health information.

[0790] A "server" is a computer system that receives and stores information sent by a user, sends requests to the generating artificial intelligence, processes the results, and provides them to the user.

[0791] A "terminal" is a device through which a user inputs information, makes requests, and receives and displays responses from a server.

[0792] "Generative AI" is an AI model that generates optimal recipes, procedures, technical information, and menus based on a user's requests and profile.

[0793] A "recipe" is a list of instructions or ingredients for cooking a particular dish.

[0794] "Real-time assistance" refers to assistance that provides the user with the next cooking steps on the spot as they proceed with their cooking.

[0795] "Professional technical information" is information about the techniques and tips of professional chefs.

[0796] A "healthy menu" refers to a meal plan that takes into consideration the user's health and nutritional balance.

[0797] The present invention is a cooking assistance system that uses interactive generative artificial intelligence to provide users with customized cooking recipes, cooking procedures, professional technical information, and healthy menus. Specific embodiments of the present invention are described below.

[0798] The system mainly consists of a server, a terminal, and a user. The server collects user information, stores and processes data, and runs generative AI models. The terminal receives input from the user, communicates with the server, and displays information to the user.

[0799] Hardware and software used

[0800] Hardware:

[0801] Server: A high-performance computer (e.g., a cloud service server)

[0802] Devices: smartphones, tablets, computers

[0803] software:

[0804] Generative AI models: such as OpenAI's GPT-3

[0805] Database: MySQL, MongoDB

[0806] Communication method: REST API, WebSocket

[0807] Front-end technologies: HTML, JavaScript, Swift

[0808] Collection of User Information

[0809] When the user first starts the app, they enter their name, age, food preferences, allergy information, and health information on the input screen on their device. The entered information is sent from the device to the server and stored in a database on the server. This creates a foundation for providing support customized for each user.

[0810] Providing cooking recipes

[0811] If a user wants to know about recipes that use a specific ingredient, they can make a request, for example, "Tell me recipes that use tomatoes." This request is sent from the device to the server, which then uses the user's profile information and the request to generate the optimal recipe for the generative AI model. The generative AI model then sends the generated recipe information back to the server, which then sends it to the device and displays it to the user.

[0812] Example: "Tomato and basil pasta"

[0813] Real-time assistance with cooking procedures

[0814] When a user actually starts cooking and wants to know the next step, they can make a request, for example, "Tell me the next step." This request is sent from the device to the server, and the server has the generative AI model generate the next step and send the result back to the server. The server then sends the step information to the device, and the device provides the step to the user via voice or text.

[0815] Example: "Please chop the tomatoes finely."

[0816] Providing professional technical information

[0817] If a user wants to know professional technical information for a particular recipe, they can make a request, for example, "Tell me the professional tips for this recipe." This request is sent from the device to the server, which queries a database of professional technical information and optimizes that information for the generative AI model. The optimized technical information is then sent from the server to the device, which then provides that information to the user.

[0818] Example: "Professionals blanch tomatoes before chopping them to enhance their flavor."

[0819] Healthy menu suggestions

[0820] If a user wants to know about healthy menus, they can make a request, for example, "Tell me about healthy menus." This request is sent from the device to the server, and the server uses the user's health checkup data and nutritional balance information to generate an optimal menu using the generative AI model. The generated menu is then sent to the device via the server and displayed to the user.

[0821] Example: "Breakfast: Oatmeal and fruit, Lunch: Grilled salmon and salad, Dinner: Chicken and vegetable soup."

[0822] As described above, the cooking assistance system of the present invention can provide users with customized recipes and procedures, professional technical information, and healthy menus, thereby providing practical and effective support.

[0823] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0824] Step 1:

[0825] Collection of User Information

[0826] Input: The user enters their name, age, food preferences, allergy information, and health information into the terminal.

[0827] Processing: The terminal sends the entered information to the server, which receives the information and processes it to save it in a database. Specifically, the data sent from the terminal undergoes validation checks on the server side before being saved in a database (for example, MySQL or MongoDB).

[0828] Output: Save to database, and user information is stored on the server.

[0829] Specific operation: When the user presses the send button, the message "Sent. Thank you." is displayed on the terminal.

[0830] Step 2:

[0831] Providing cooking recipes

[0832] Input: The user types a request into the terminal, such as "Tell me some recipes using tomatoes."

[0833] Processing: The device sends this request to the server, which then sends the received request and the user's profile information to the generative AI model (e.g., OpenAI's GPT-3). The generative AI model generates the optimal recipe based on the prompt, which includes the user's profile information (such as name, age, and food preferences).

[0834] Output: The recipe information generated by the generative AI model is sent back to the server, which then sends it to the device and displays it to the user.

[0835] Specific operation: An image of "Tomato and Basil Pasta" and recipe instructions will be displayed on the device screen.

[0836] Step 3:

[0837] Real-time assistance with cooking procedures

[0838] Input: The user types a request into the terminal, such as "Tell me what to do next."

[0839] Processing: The device sends a request to the server, which then sends it to the generative AI model to generate the next step. The prompt contains the current step and the next step. The generative AI model generates the next step and sends the result back to the server.

[0840] Output: The generated next step information is sent from the server to the terminal and provided to the user by voice or text.

[0841] Specific operation: The device will say, "Please finely chop the tomatoes."

[0842] Step 4:

[0843] Providing professional technical information

[0844] Input: A user types a request into a device, such as "Tell me some pro tips for this recipe."

[0845] Processing: The device sends a request to the server, which queries a professional technical information database to obtain information and optimizes it for the generative AI model. The prompt contains the recipe and related technical information. The generative AI model returns the optimized technical information to the server.

[0846] Output: The optimized technical information is sent from the server to the terminal and displayed or explained to the user by voice.

[0847] Specific action: The information displayed is, "Professionals blanch tomatoes before chopping them to make them taste even better."

[0848] Step 5:

[0849] Healthy menu suggestions

[0850] Input: The user inputs a request into the terminal, such as "Tell me some healthy meals."

[0851] Processing: The device sends a request to the server, which then sends a prompt to the generative AI model to generate an optimal menu based on the user's health checkup data and nutritional balance information. The generative AI model generates a menu that takes nutritional balance into consideration and returns the results to the server.

[0852] Output: The generated menu information is sent from the server to the terminal and displayed to the user.

[0853] Specific behavior: The menu will be displayed: "Breakfast: Oatmeal and fruit, Lunch: Grilled salmon and salad, Dinner: Chicken and vegetable soup."

[0854] (Application example 1)

[0855] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0856] Currently, there are a wide variety of systems that provide cooking instructions and recipes, but these systems generally do not take into account the user's individual preferences, health status, or the specific ingredients used. Even in brick-and-mortar stores, customers are rarely offered recipe suggestions based on the ingredients they have on hand, and it is difficult to obtain professional technical advice or next cooking steps in real time. As a result, users often have to go through a lot of effort to obtain the information they want, resulting in a lack of convenience.

[0857] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0858] In this invention, the server includes a means for inputting user information and transmitting it to the server, a means for saving the input user information, and a means for causing a recipe generation AI to generate a recipe based on designated ingredients and providing the recipe to the user. This makes it possible to propose recipes based on ingredients in-store using advanced information terminals. It also makes it possible to provide the user with next steps in real time by voice or text.

[0859] Furthermore, by providing a means to acquire professional technical information, optimize it, and provide it to users, it is possible to convey professional cooking techniques and tips to users. In addition, by using health-related information to generate healthy menus using artificial intelligence, it is possible to provide these menus to users, making it possible to suggest optimal menus based on each individual's health status. This will greatly improve user convenience and provide optimal cooking support.

[0860] "Interactive generative artificial intelligence" is an artificial intelligence that collects information through dialogue with users and generates and provides appropriate information and services based on that information.

[0861] A "cooking assistance system" is a system that assists users when cooking by suggesting recipes and guiding them through cooking procedures.

[0862] "User Information" refers to personal information such as the user's name, age, food preferences, allergy information, and health information.

[0863] A "server" is a computer system used to store user information, process data, and generate recipes.

[0864] "Advanced information terminals" refer to electronic devices with advanced functions, such as smartphones and digital kiosks.

[0865] A "recipe" is a collection of information on how to make a dish, such as the ingredients to be used, their amounts, and cooking steps.

[0866] "Professional technical information" refers to information related to cooking techniques and tips possessed by professional chefs.

[0867] "Health-related information" refers to data related to maintaining and improving health, such as a user's health checkup data and nutritional balance information.

[0868] A "menu" refers to a meal combination or menu plan for a specific period of time.

[0869] The present invention relates to a cooking assistance system that uses interactive generative artificial intelligence and aims to provide optimal assistance to users when cooking. Specific embodiments will be described below.

[0870] 1. System Configuration

[0871] This system consists of a terminal for inputting user information, a server for storing and processing data, and multiple functions that utilize the generative AI model. The terminals are advanced information terminals such as smartphones and digital kiosks. The server stores user information and processes the data, and generates various data using a generative AI model (e.g., OpenAI's GPT-4).

[0872] 2. Collection of User Information

[0873] The server collects user information (such as name, age, food preferences, allergy information, and health information) sent from the device and stores it in a database, enabling it to provide customized services tailored to each individual user.

[0874] 3. Recipe Generation

[0875] When a user wants to know a recipe based on specific ingredients, the device sends the request to the server. The server uses the user information and the request to generate the optimal recipe using a generative AI model, which then sends the recipe back to the device. For example, a recipe such as "pasta with tomatoes and basil" is generated and served to the user.

[0876] 4. Providing real-time procedures

[0877] When a user wants real-time instructions for cooking a dish, the device sends a request to the server, which uses a generative AI model to generate the next steps and sends them back to the device in real time, providing specific instructions via voice or text, such as "finely chop the tomatoes."

[0878] 5. Providing professional technical information

[0879] When a user wants to know professional technical information for a particular recipe, the device sends the request to the server. The server queries a database of professional technical information, optimizes the information with a generative AI model, and provides it to the user. For example, the information provided may be, "Professionals recommend blanching the tomatoes before chopping them to enhance the flavor."

[0880] 6. Healthy meal suggestions

[0881] When a user wants to know about healthy meals, the device sends a request to the server. The server uses the user's health checkup data and nutritional balance information to generate an optimal meal plan using the generative AI model, and sends the plan back to the device. For example, a menu such as "Breakfast: Oatmeal and fruit, Lunch: Grilled salmon and salad, Dinner: Chicken and vegetable soup" may be provided.

[0882] Hardware and Software

[0883] The server is built using programming languages ​​and frameworks such as Python and Flask, and uses OpenAI's API for generative AI, with user information stored in a data storage system such as an SQLite database.

[0884] Specific examples

[0885] Examples of prompts include:

[0886] "Tell me a recipe that uses tomatoes, basil, and pasta."

[0887] "Tell me the next cooking step"

[0888] "What are some pro tips for this recipe?"

[0889] By using these prompts, users can obtain optimal recipes, cooking procedures, and specialized technical information in real time. In this way, a cooking assistance system using interactive generative AI can provide users with highly convenient and effective support.

[0890] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0891] Step 1:

[0892] Collection of User Information

[0893] Users enter their name, age, food preferences, allergy information, and health information through a device (smartphone or digital kiosk). The entered data is sent from the device to a server, which receives the data and stores it in an SQLite database. This creates the foundation for providing services customized for each user.

[0894] Step 2:

[0895] Recipe Generation

[0896] When a user wants to know a recipe based on a specific ingredient, they can input, for example, "Tell me a recipe using tomatoes" into their device. The device then sends this request to the server. The server references the user information and the request and generates the optimal recipe based on a generative AI model (for example, GPT-4). The generated recipe is then sent back from the server to the device and displayed to the user. Specifically, in response to the prompt "Tell me a recipe using tomatoes, basil, and pasta," "Pasta with tomatoes and basil" is suggested.

[0897] Step 3:

[0898] Providing real-time procedures

[0899] When a user wants real-time support for cooking steps, they can type, for example, "Tell me the next steps" into their device. Upon receiving this request, the device sends a request to the server. The server generates the next cooking steps based on a generative AI model. These steps are sent back from the server to the device and provided to the user via text or voice. Specifically, instructions such as "Please finely chop the tomatoes" are provided in real time.

[0900] Step 4:

[0901] Providing professional technical information

[0902] If a user wants to know professional technical information for a particular recipe, they can type, for example, "Tell me the pro tips for this recipe" into their device. The device then sends this request to the server. The server queries a database of professional technical information and optimizes the information using a generative AI model. The optimized information is then sent back from the server to the device and presented to the user via display or audio. For example, they might provide information such as, "Professionals blanch tomatoes before chopping them to enhance their flavor."

[0903] Step 5:

[0904] Healthy menu suggestions

[0905] If a user wants to know what a healthy menu is, they can type, for example, "Tell me what a healthy menu is." The device sends this request to the server. The server uses the user's health checkup data and nutritional balance information to have the generative AI model generate the optimal menu. This menu information is sent back from the server to the device and provided to the user. Specifically, the suggested menu might be "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup."

[0906] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0907] The present invention is a cooking assistance system that utilizes an emotion engine in addition to interactive generative artificial intelligence, and provides more personalized assistance by recognizing the user's emotions and customizing recipes and procedures based on those emotions, and adjusting the tone and content of the dialogue. Specific embodiments of the present invention are described below.

[0908] System Overview

[0909] Users can interact with the system through an interface and receive various cooking assistance. The system utilizes information from the server, the device, and the user, and also uses an emotion engine to suggest recipes, provide real-time assistance with procedures, provide professional technical information, and suggest healthy meals.

[0910] Program processing overview

[0911] The system works through user information collection, recipe generation, real-time assistance, professional technical information provision, healthy meal suggestions, as well as emotion recognition and optimization using an emotion engine.

[0912] Collection of User Information

[0913] 1. The user launches the app for the first time and is prompted to enter user information.

[0914] 2. The device displays fields for the user to enter their name, age, food preferences, allergy information, and health information.

[0915] 3. The user enters this information and presses the submit button.

[0916] 4. The terminal sends the entered information to the server.

[0917] 5. The server stores the received user information in a database.

[0918] Providing cooking recipes

[0919] 1. A user voices or texts a request for a recipe using a specific ingredient (e.g., "Tell me some recipes using tomatoes").

[0920] 2. The terminal sends the input request to the server.

[0921] 3. The server uses the AI ​​to generate the optimal recipe based on the user's profile information and request.

[0922] 4. The generation AI returns the generated recipe information to the server.

[0923] 5. The server sends the recipe information to the terminal, and the terminal displays the generated recipe (e.g., "Pasta with Tomato and Basil") for the user.

[0924] Real-time assistance with cooking procedures

[0925] 1. If a user is cooking and wants to know the next step, they can send a request via voice or text (e.g., "Tell me the next step").

[0926] 2. The device sends a request to the server.

[0927] 3. The server has the generation AI generate the next steps and send those steps back to the server.

[0928] 4. The generative AI generates next step information (e.g., "Please finely chop the tomatoes").

[0929] 5. The server receives the generated procedure information and sends it to the terminal.

[0930] 6. The device will provide the user with next steps via voice or text.

[0931] Providing professional technical information

[0932] 1. A user submits a request for pro technical information on a recipe (e.g., "What are some pro tips for this recipe?").

[0933] 2. The device sends a request to the server.

[0934] 3. The server queries a professional technical information database and provides that information to the generation AI.

[0935] 4. Generative AI optimizes technical information and generates specific advice (e.g., "Professionals recommend blanching tomatoes before chopping them to make them taste even better").

[0936] 5. The server receives the optimized technical information and sends it to the device.

[0937] 6. The device displays or audibly explains technical information to the user.

[0938] Healthy menu suggestions

[0939] 1. If a user wants to know about healthy meals, they send a request (e.g., "Tell me about healthy meals").

[0940] 2. The device sends a request to the server.

[0941] 3. The server retrieves the user's health checkup data and nutritional balance information from the database and provides it to the generation AI.

[0942] 4. Generative AI generates nutritionally balanced menus based on health information (e.g., "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup").

[0943] 5. The server receives the generated menu information and sends it to the terminal.

[0944] 6. The device displays the menu information to the user.

[0945] Utilizing the Emotion Engine

[0946] 1. If the user's emotions change during a conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion.

[0947] 2. The device sends the recognized emotion data to the server.

[0948] 3. The server provides the generative AI with emotional data and issues instructions to customize the recipe, steps, and dialogue content.

[0949] For example, if a user is feeling stressed about cooking, the emotion engine will recognize this and the generative AI will suggest recipes that are easy and quick to make. This allows users to receive optimal cooking assistance tailored to their mood and situation.

[0950] In this way, a cooking assistance system that combines interactive generative artificial intelligence and an emotion engine can provide users with customized recipes and procedures, professional techniques, and healthy menus, and can also respond to their emotions, providing more personalized assistance.

[0951] The processing flow will be explained below.

[0952] Processing flow of a cooking support system using an emotion engine

[0953] Collection of User Information

[0954] Step 1:

[0955] The user launches the app for the first time and is presented with a screen to enter user information.

[0956] Step 2:

[0957] The device will present the user with fields to enter their name, age, food preferences, allergy information, and health information.

[0958] Step 3:

[0959] The user enters this information and presses the send button.

[0960] Step 4:

[0961] The terminal transmits the input information to the server.

[0962] Step 5:

[0963] The server stores the received user information in a database.

[0964] Providing cooking recipes

[0965] Step 1:

[0966] The user voices or types in a request for a recipe using a specific ingredient (e.g., "Tell me some recipes using tomatoes").

[0967] Step 2:

[0968] The terminal transmits the input request to the server.

[0969] Step 3:

[0970] The server references the user's profile information from the database and sends the request to the generation artificial intelligence.

[0971] Step 4:

[0972] Generative AI generates the optimal recipe based on the user's profile and specified ingredients.

[0973] Step 5:

[0974] The server receives the generated recipe information and transmits it to the terminal.

[0975] Step 6:

[0976] The device displays the generated recipe (e.g., "Pasta with Tomato and Basil") to the user.

[0977] Real-time assistance with cooking procedures

[0978] Step 1:

[0979] If a user is cooking and wants to know the next step, they can send a request by voice or text (e.g., "Tell me the next step").

[0980] Step 2:

[0981] The device sends a request to the server.

[0982] Step 3:

[0983] The server issues instructions to the generation artificial intelligence to generate the next step.

[0984] Step 4:

[0985] Generative AI generates next step information (e.g., "Please finely chop the tomatoes").

[0986] Step 5:

[0987] The server receives the generated procedure information and transmits it to the terminal.

[0988] Step 6:

[0989] The device will provide the user with next steps via voice or text.

[0990] Providing professional technical information

[0991] Step 1:

[0992] A user submits a request for pro technical information on a recipe (e.g., "What are some pro tips for this recipe?").

[0993] Step 2:

[0994] The device sends a request to the server.

[0995] Step 3:

[0996] The server queries a professional technical information database and provides the information to the generating artificial intelligence.

[0997] Step 4:

[0998] Generative AI optimizes technical information and generates specific advice (e.g., "Professionals recommend blanching tomatoes before chopping them to make them taste even better").

[0999] Step 5:

[1000] The server receives the optimized technical information and sends it to the terminal.

[1001] Step 6:

[1002] The terminal displays or audibly explains technical information to the user.

[1003] Healthy menu suggestions

[1004] Step 1:

[1005] If a user wants to know about healthy meals, they send a request (e.g., "Tell me about healthy meals").

[1006] Step 2:

[1007] The device sends a request to the server.

[1008] Step 3:

[1009] The server retrieves the user's health checkup data and nutritional balance information from the database and provides it to the generating artificial intelligence.

[1010] Step 4:

[1011] Generative artificial intelligence generates nutritionally balanced menus based on health information (e.g., "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup").

[1012] Step 5:

[1013] The server receives the generated menu information and transmits it to the terminal.

[1014] Step 6:

[1015] The terminal displays the menu information to the user.

[1016] Utilizing the Emotion Engine

[1017] Step 1:

[1018] If the user's emotions change during the conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion.

[1019] Step 2:

[1020] The device transmits the recognized emotion data to the server.

[1021] Step 3:

[1022] The server provides emotional data to the generative AI, which then issues instructions to customize recipes, steps, and dialogue content.

[1023] As a concrete example, if a user is feeling stressed about cooking, the emotion engine will recognize this and the generative AI will suggest recipes that are easy and quick to make. In this example, it is possible to suggest "time-saving recipes that can be made in the microwave." This allows users to receive optimal cooking support tailored to their mood and situation.

[1024] Example 2

[1025] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1026] In today's modern lifestyles, it is extremely important to provide healthy and efficient cooking assistance that accommodates the different lifestyles and food preferences of each individual user. However, conventional cooking assistance systems lack the ability to customize based on the user's emotions and health status, making it difficult to provide more personalized assistance. There is also a need to effectively incorporate technical information from experts to improve the quality of cooking. Furthermore, in situations where real-time response is required, providing prompt and appropriate information has been insufficient.

[1027] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1028] In this invention, the server includes means for inputting user information and transmitting it to the server, means for saving the input user information, means for having a generation AI generate a recipe based on specified ingredients and providing the recipe to the user, means for providing the user with real-time audio or text instructions for the next steps, means for acquiring technical information from experts and optimizing it to provide to the user, means for having the generation AI generate a healthy menu based on health-related information and providing the menu to the user, and means for analyzing the user's emotions using an emotion recognition engine and customizing the recipe, steps, and dialogue content based on the emotions. This enables personalized cooking support according to the individual needs and emotions of the user, provision of menus that accommodate health management, and high-quality cooking support utilizing the skills of experts.

[1029] "User information" refers to the name, age, food preferences, allergy information, health information, etc. entered by the user.

[1030] "Server" refers to a computer system that stores input user information, processes various data, and provides it to the generation AI.

[1031] "Specified ingredients" refers to ingredients that the user specifies to be used in creating the recipe.

[1032] "Generative AI" refers to an AI model that generates recipes, steps, menus, etc. based on user requests.

[1033] "Real-time" refers to nearly immediate response to user requests.

[1034] "Expert technical information" refers to information about professional knowledge and techniques related to cooking.

[1035] "Health-related information" refers to all data related to health and nutrition, such as a user's health checkup data and nutritional balance information.

[1036] An "emotion recognition engine" refers to a system that recognizes emotions by analyzing the user's voice and facial expressions.

[1037] "Personalized cooking assistance" refers to providing cooking assistance customized based on the user's individual information and emotions.

[1038] "Customizing dialogue content" refers to adjusting the tone and content of dialogue provided according to the user's emotions and situation.

[1039] The present invention is a cooking assistance system that utilizes interactive generative artificial intelligence (generative AI model) and an emotion engine to recognize the user's emotions and customize recipes and procedures according to those emotions, thereby providing more personalized assistance. A specific embodiment of this system is shown below.

[1040] Overall system overview

[1041] Users can interact with the system through devices such as smartphones and tablets and receive various cooking assistance. The system utilizes information from the server, devices, and users, and uses an emotion engine to suggest recipes, provide real-time assistance with procedures, provide expert technical information, and suggest healthy meals. The system mainly uses the following hardware and software:

[1042] Hardware: smartphones, tablets, servers, database servers

[1043] Software: Interactive generative AI models, emotion recognition engines, user interface applications, database management systems

[1044] Program processing overview

[1045] The system works through user information collection, recipe generation, real-time assistance, expert technical input, healthy meal suggestions, as well as emotion recognition and optimization using an emotion engine.

[1046] Collection of User Information

[1047] When a user launches the app for the first time, a screen for entering user information is displayed. The device displays fields for entering the user's name, age, food preferences, allergy information, and health information. When the user enters this information and presses the send button, the device sends the entered information to the server. The server stores the received user information in a database.

[1048] Providing cooking recipes

[1049] A user inputs a request for a recipe using a specific ingredient via voice or text. For example, if the request is "Tell me a recipe using tomatoes," the device analyzes the input request and sends it to the server. The server uses the user's profile information and the request content to have the generative AI model generate the optimal recipe. The generative AI generates recipe information, such as "pasta with tomatoes and basil," and sends it back to the server. The server then sends the generated recipe information to the device and displays it to the user.

[1050] Real-time assistance with cooking procedures

[1051] When a user wants to know the next step while cooking, they send a request by voice or text. For example, if a request is made to "tell me the next step," the device sends the request to the server. The server then has the generative AI model generate the next step and sends that step back to the server. The generative AI generates step information such as "finely chop the tomatoes," and the server sends that information to the device. The device then provides the next step to the user by voice or text.

[1052] Providing technical information from experts

[1053] A user sends a request for expert technical information about a recipe. For example, if the user requests, "Tell me the pro tips for this recipe," the device sends the request to the server. The server queries a database of expert technical information and provides that information to the generative AI model. The generative AI generates specific technical information, such as "Professionals recommend blanching tomatoes before chopping them to enhance their flavor," and sends it back to the server. The server then sends that information to the device and explains it to the user by displaying it or by voice.

[1054] Healthy menu suggestions

[1055] When a user wants to know about healthy menu options, they send a request. For example, if the request is "Tell me about healthy menu options," the device sends the request to the server. The server retrieves the user's health checkup data and nutritional balance information from a database and provides this to the generative AI model. The generative AI generates a healthy menu option such as "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup." The server sends the generated menu information to the device and displays it to the user.

[1056] Utilizing the Emotion Engine

[1057] If the user's emotions change during a conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion. The device then sends the recognized emotion data to the server, which then provides the emotion data to the generative AI model and issues instructions to customize the recipe, steps, and conversation content. For example, if the user is feeling stressed about cooking, the emotion engine will recognize this, and the generative AI model will suggest a recipe that is easy and quick to make. This allows the user to receive optimal cooking assistance tailored to their mood and situation.

[1058] Examples of prompt statements

[1059] Recipe generation prompt: "Generate a quick recipe using tomatoes. The user is stressed."

[1060] Technical information prompt: "What are some pro tips for this recipe?"

[1061] In this way, the cooking assistance system of the present invention combines interactive generative artificial intelligence with an emotion engine to provide users with customized recipes, procedures, expert techniques, and healthy menus, and can also respond to emotions, making it possible to provide more personalized assistance.

[1062] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1063] Collection of User Information

[1064] Step 1:

[1065] The user launches the app for the first time and is presented with a screen to enter user information.

[1066] What it does: The device displays fields for entering name, age, food preferences, allergy information, and health information.

[1067] Step 2:

[1068] The user enters this information and presses the send button.

[1069] Specific operation: The device receives the entered information, detects when the send button is pressed, and sends it to the server.

[1070] Input: User's personal information (name, age, food preferences, allergy information, health information)

[1071] Output: User information sent to the server

[1072] Step 3:

[1073] The terminal transmits the input information to the server.

[1074] Specific operation: The terminal sends the received user information to the server via a secure communication channel.

[1075] Input: User information (secure format)

[1076] Output: User information sent to the server

[1077] Step 4:

[1078] The server stores the received user information in a database.

[1079] Specific operation: The server stores the received user information in a database and makes it available for subsequent processes.

[1080] Input: User information sent to the server

[1081] Output: User information stored in the database

[1082] Providing cooking recipes

[1083] Step 1:

[1084] A user voices or texts a request for a recipe using a specific ingredient.

[1085] Specific action: For example, request "Tell me some recipes that use tomatoes."

[1086] Input: Request for a specific material

[1087] Output: The text or audio data of the request

[1088] Step 2:

[1089] The terminal transmits the input request to the server.

[1090] Specific operation: The device performs voice recognition and text analysis and sends the request content to the server.

[1091] Input: The text or audio data of the request

[1092] Output: The request sent to the server

[1093] Step 3:

[1094] The server generates a prompt for the generative AI model based on the user's profile information and request content.

[1095] Specific operation: The server converts the user information and request content into a prompt text.

[1096] Input: User profile information and request details

[1097] Output: Prompt sentence for generative AI model

[1098] Step 4:

[1099] The generative AI model generates recipe information based on the prompt sentence and sends it back to the server.

[1100] Specific behavior: The generative AI generates specific recipes such as "pasta with tomato and basil."

[1101] Input: prompt statement

[1102] Output: Generated recipe information

[1103] Step 5:

[1104] The server transmits the generated recipe information to the terminal.

[1105] Specific operation: The server transfers the recipe information to the device.

[1106] Input: Generated recipe information

[1107] Output: Recipe information sent to the device

[1108] Step 6:

[1109] The terminal displays the generated recipe to the user.

[1110] Specific operation: The device displays the received recipe information on the user interface.

[1111] Input: Recipe information sent to the device

[1112] Output: Recipe information displayed to the user

[1113] Real-time assistance with cooking procedures

[1114] Step 1:

[1115] If a user is cooking and wants to know the next step, they can send a request by voice or text.

[1116] Specific behavior: For example, request "Tell me what the next step is."

[1117] Input: Next Step Request

[1118] Output: The text or audio data of the request

[1119] Step 2:

[1120] The device sends a request to the server.

[1121] Specific operation: The device performs voice recognition or text analysis and forwards the request content to the server.

[1122] Input: The text or audio data of the request

[1123] Output: The request sent to the server

[1124] Step 3:

[1125] The server asks the generative AI model to generate the next step as a prompt.

[1126] Specific operation: The server summarizes the current procedure information into a prompt sentence and passes it to the generative AI model.

[1127] Input: Current procedure information

[1128] Output: Prompt sentence for generative AI model

[1129] Step 4:

[1130] The generative AI model generates next step information and sends it back to the server.

[1131] Specific actions: The generative AI generates specific steps such as "finely chop the tomatoes."

[1132] Input: prompt statement

[1133] Output: Generated procedure information

[1134] Step 5:

[1135] The server transmits the generated procedure information to the terminal.

[1136] Specific operation: The server transfers the generated procedure information to the terminal.

[1137] Input: Generated procedure information

[1138] Output: Instructions sent to the terminal

[1139] Step 6:

[1140] The device will provide the user with next steps via voice or text.

[1141] Specific operation: The device explains the received procedure information to the user by voice or text.

[1142] Input: Instructions sent to the terminal

[1143] Output: Next steps information provided to the user

[1144] Providing technical information from experts

[1145] Step 1:

[1146] A user submits a request for expert technical information on a recipe.

[1147] Specific behavior: For example, request "Tell me some pro tips for this recipe."

[1148] Input: Technical Information Request

[1149] Output: The text or audio data of the request

[1150] Step 2:

[1151] The device sends a request to the server.

[1152] Specific operation: The device performs voice recognition or text analysis and forwards the request content to the server.

[1153] Input: The text or audio data of the request

[1154] Output: The request sent to the server

[1155] Step 3:

[1156] The server queries a database of technical information from experts.

[1157] Specific operation: The server refers to the database based on the request and retrieves technical information.

[1158] Input: Request details

[1159] Output: Expert technical information

[1160] Step 4:

[1161] The generative AI model generates specific, optimized advice based on technical information and sends it back to the server.

[1162] Specific operation: The generative AI generates advice such as, "Professionals recommend blanching tomatoes before chopping them to make them taste even better."

[1163] Input: Expert technical information

[1164] Output: The generated advice

[1165] Step 5:

[1166] The server transmits the generated advice to the terminal.

[1167] Specific operation: The server transfers the generated advice to the terminal.

[1168] Input: Generated advice

[1169] Output: Advice sent to terminal

[1170] Step 6:

[1171] The terminal displays or audibly explains the advice to the user.

[1172] Specific operation: The device displays the received advice on the user interface or explains it aloud.

[1173] Input: Advice sent to terminal

[1174] Output: Advice given to the user

[1175] Healthy menu suggestions

[1176] Step 1:

[1177] If a user wants to know about healthy meals, they send a request.

[1178] Specific action: For example, request "Tell me some healthy meals."

[1179] Input: Healthy Meal Request

[1180] Output: The text or audio data of the request

[1181] Step 2:

[1182] The device sends a request to the server.

[1183] Specific operation: The device performs voice recognition or text analysis and forwards the request content to the server.

[1184] Input: The text or audio data of the request

[1185] Output: The request sent to the server

[1186] Step 3:

[1187] The server retrieves the user's health checkup data and nutritional balance information from the database.

[1188] Specific operation: The server retrieves data related to the user's health from the database.

[1189] Input: User's health checkup data, nutritional balance information

[1190] Output: Health data collected by the server

[1191] Step 4:

[1192] The generative AI model generates menus that take nutritional balance into consideration based on health information.

[1193] Specific operation: The generative AI generates menus such as "Breakfast: oatmeal and fruit, Lunch: grilled salmon and salad, Dinner: chicken and vegetable soup."

[1194] Input: User's health checkup data and nutritional balance information

[1195] Output: Generated healthy meal plan

[1196] Step 5:

[1197] The server transmits the generated menu information to the terminal.

[1198] Specific operation: The server transfers the generated menu information to the terminal.

[1199] Input: Generated menu information

[1200] Output: Menu information sent to the device

[1201] Step 6:

[1202] The terminal displays the menu information to the user.

[1203] Specific operation: The device displays the received menu information on the user interface.

[1204] Input: Menu information sent to the device

[1205] Output: Menu information provided to the user

[1206] Utilizing the Emotion Engine

[1207] Step 1:

[1208] If the user's emotions change during the conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion.

[1209] How it works: The emotion engine analyzes the user's voice and facial expressions in real time to detect emotions such as stress or joy.

[1210] Input: User voice and facial expression data

[1211] Output: Recognized emotion data

[1212] Step 2:

[1213] The device transmits the recognized emotion data to the server.

[1214] Specific operation: The device sends emotion data to the server.

[1215] Input: Recognized emotion data

[1216] Output: Emotion data sent to the server

[1217] Step 3:

[1218] The server provides emotional data to a generative AI model, which then provides instructions to customize recipes, steps, and dialogue.

[1219] Specific operation: The server incorporates emotional data into a prompt sentence and passes it to the generative AI model.

[1220] Input: Emotion data

[1221] Output: Prompt sentence for generative AI model

[1222] Step 4:

[1223] The generative AI model generates emotion-based recipes and instructions and sends them back to the server.

[1224] For example, if a user is feeling stressed about cooking, the generative AI will generate a recipe such as, "If you're feeling stressed, make tomato and basil pasta. It's easy to make."

[1225] Input: prompt statement

[1226] Output: The generated recipe or instructions

[1227] Step 5:

[1228] The server sends the generated recipes and instructions to the device.

[1229] Specific operation: The server transfers the generated information to the terminal.

[1230] Input: Generated recipes and instructions

[1231] Output: Information sent to the terminal

[1232] Step 6:

[1233] The device displays or audibly explains the recipe and steps to the user.

[1234] Specific operation: The device displays or audibly explains the received information.

[1235] Input: Information sent to the device

[1236] Output: The recipe or instructions provided to the user

[1237] (Application example 2)

[1238] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1239] Conventional cooking assistance systems and production process management systems did not provide support that took into account the emotions of users and workers. As a result, they were unable to respond appropriately to situations that caused stress to users and workers, leading to problems such as reduced work efficiency and decreased motivation. In addition, it was difficult to respond to the individual needs of users and workers, resulting in a lack of personalized support.

[1240] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting user information and transmitting it to the server, means for saving the input user information, means for having a generation AI generate a recipe based on specified ingredients and providing the recipe to the user, means for providing the user with real-time audio or text instructions on how to proceed, means for acquiring professional technical information, optimizing it, and providing it to the user, means for collecting emotion data using an emotion engine that recognizes the user's emotions and customizing the content of the provision based on that emotion data, means for having a generation AI generate optimal support information based on the emotion data and work content of factory workers and providing it to the workers, and means for having a generation AI generate healthy menus based on health-related information and providing the menus to the user. This provides personalized support that takes the emotions of users and workers into consideration, thereby reducing stress and improving work efficiency.

[1241] "Interactive generative artificial intelligence" is an artificial intelligence that collects information through dialogue with the user and makes appropriate suggestions and assistance based on that information.

[1242] A "system" is a collection of devices and software in which multiple elements work together to achieve a specific function.

[1243] "User Information" means certain data about you, such as your name, age, food preferences, allergy information, and health information.

[1244] A "server" is a computer system that provides services and data to other computers on a network.

[1245] An "emotion engine" is software that analyzes the user's voice and facial expressions to recognize emotions and process information accordingly.

[1246] A "recipe" is a list of steps and ingredients for making a particular dish.

[1247] "Professional technical information" refers to specific techniques and know-how provided by experts.

[1248] A "healthy menu" is a combination of nutritionally balanced meals.

[1249] A "factory work support system" is a system that aims to improve the efficiency of work within a factory and increase quality and productivity.

[1250] "Workers" are workers who work in factories and other places.

[1251] "Emotion data" refers to information about the emotions of users and workers obtained by the emotion engine.

[1252] This invention is a factory work support system that uses interactive generative artificial intelligence and an emotion engine, and provides personalized support according to the emotions of users and workers.

[1253] System Overview

[1254] This system includes a terminal, a server, an emotion engine, and a generative AI model. The terminal provides an interface for users and workers to input information, and the server stores and processes the input information and generates appropriate support information.

[1255] Hardware and Software

[1256] 1. Hardware:

[1257] Cameras for facial recognition (e.g., high-performance webcams)

[1258] Microphone for voice analysis (e.g. high-sensitivity microphone)

[1259] Terminals for workers (e.g., tablet terminals)

[1260] 2. Software:

[1261] Emotion recognition libraries (e.g., emotion analysis APIs)

[1262] Generative AI models (e.g., interactive generative AI engines)

[1263] Database (e.g. SQL database)

[1264] Front-end (e.g., web browser-based interface)

[1265] Data processing and calculation

[1266] The server receives the user information and worker information sent from the terminal and performs the following data processing and calculations.

[1267] 1. Emotion recognition:

[1268] Cameras and microphones are used to collect the facial expressions and voices of workers.

[1269] Using the sentiment analysis API, the collected data is analyzed to obtain sentiment data.

[1270] 2. Generating support information:

[1271] Based on emotion data and task data, the generative AI model generates appropriate work procedures and support information.

[1272] The generated information is stored in a database and transmitted to the terminal as needed.

[1273] Specific use cases

[1274] Let's say a worker is performing assembly work in a factory. While working, a camera and microphone analyze the worker's facial expressions and voice, and an emotion engine detects stress. The server receives the stress data and sends a prompt to the generation AI saying, "The worker is in a stressful state, so please suggest a simplified procedure for the work." The generation AI generates a simplified procedure, and the server sends that information to the terminal, presenting the simple procedure to the worker.

[1275] Prompt Sentence Examples

[1276] "High Stress" state. Please simplify the current task and suggest next steps.

[1277] In this way, a factory work support system that combines interactive generative artificial intelligence and an emotion engine can provide flexible support that responds to the emotions of workers, improving work efficiency and worker satisfaction.

[1278] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1279] Step 1:

[1280] The terminal provides an interface for inputting user information. The user or worker inputs user information such as name, age, food preferences, allergy information, and health information, and presses the send button. The input data is sent to the server.

[1281] Step 2:

[1282] The server stores the received user information in a database, including user and worker profile information.

[1283] Step 3:

[1284] A request for cooking or task assistance is entered via voice or text from the device. For example, a request might be "Tell me a recipe using tomatoes" or "Tell me the next step in the task." The request is then sent to the server.

[1285] Step 4:

[1286] The server sends a prompt to the generative AI model based on the request content and user information to generate the optimal recipe and work procedure. An example of a prompt sentence is, "The worker is under stress, so please suggest a procedure that simplifies the work."

[1287] Step 5:

[1288] A generative AI model generates recipes and instructions based on the prompt, which are then sent back to the server. Examples include "pasta with tomatoes and basil" and instructions such as "finely chop the tomatoes."

[1289] Step 6:

[1290] The server sends the information received from the generation AI to the terminal, which then displays or speaks the generated recipes and work procedures to the user or worker.

[1291] Step 7:

[1292] The device's camera and microphone analyze the facial expressions and voices of users and workers in real time. The emotion engine recognizes emotions based on the analyzed data and sends specific emotional data to the server.

[1293] Step 8:

[1294] Based on the emotion data received by the server, the generative AI model is prompted again to generate assistance according to the emotion, resulting in the generation of customized recipes and work procedures according to the emotion.

[1295] Step 9:

[1296] The server sends the regenerated information to the terminal, which then presents the user or worker with optimized recipes and work procedures, reducing stress and enabling more efficient work.

[1297] The above processing steps realize a factory work support system that utilizes interactive generative AI and an emotion engine. The collaboration between the generative AI model and the emotion engine makes it possible to provide personalized support according to the emotions of users and workers.

[1298] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1299] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1300] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1301] [Third embodiment]

[1302] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1303] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1304] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1305] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1306] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1307] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1308] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1309] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1310] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1311] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1312] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1313] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1314] The present invention provides a cooking assistance system that utilizes interactive generative artificial intelligence. Specific embodiments of the system will be described below.

[1315] System Overview

[1316] Users can interact with the system through an interface and receive various cooking assistance. The system primarily utilizes information collected from the server, terminals, and users to suggest recipes, provide real-time assistance with procedures, provide professional technical information, and suggest health-based menus.

[1317] Program processing overview

[1318] The system operates through the following steps: collecting user information, generating recipes, providing real-time assistance, providing professional technical information, and suggesting healthy meals.

[1319] Collection of User Information

[1320] 1. When a user uses the system for the first time, the terminal displays a screen for the user to enter their name, age, food preferences, allergy information, and health information.

[1321] 2. The user enters this information and presses the submit button.

[1322] 3. The device sends the entered information to the server.

[1323] 4. The server stores the received user information in a database.

[1324] 5. This allows the system to build a foundation for providing customized assistance to each user.

[1325] Providing cooking recipes

[1326] 1. If a user wants to know recipes that use a specific ingredient, they can request, for example, "Tell me recipes that use tomatoes."

[1327] 2. The device receives this request and sends it to the server.

[1328] 3. The server uses the AI ​​to generate the optimal recipe based on the user's profile information and request.

[1329] 4. The generation AI returns the generated recipe information to the server.

[1330] 5. The server sends the recipe information to the terminal, and the terminal displays the recipe to the user.

[1331] As a specific example, a recipe such as "tomato and basil pasta" is generated and provided to the user.

[1332] Real-time assistance with cooking procedures

[1333] 1. When the user actually starts cooking and wants to know the next steps, they can request, for example, "Tell me the next steps."

[1334] 2. The device sends this request to the server.

[1335] 3. The server has the generation AI generate the next steps and send those steps back to the server.

[1336] 4. The server sends the procedure information to the terminal, and the terminal provides the procedure to the user by voice or text.

[1337] For example, instructions such as "Please finely chop the tomatoes" are provided in real time.

[1338] Providing professional technical information

[1339] 1. If a user wants to know professional technical information about a particular recipe, they can request, for example, "Tell me the pro tips for this recipe."

[1340] 2. The device sends this request to the server.

[1341] 3. The server queries a professional technical information database and optimizes the information for the generating AI.

[1342] 4. The generating AI provides optimized technical information to the server.

[1343] 5. The server sends the technical information to the terminal, and the terminal displays or audibly explains the technical information to the user.

[1344] As a specific example, information such as "Professionals blanch tomatoes before chopping them to make them taste even better" is provided.

[1345] Healthy menu suggestions

[1346] 1. If a user wants to know about healthy meals, they can request, for example, "Tell me about healthy meals."

[1347] 2. The device sends this request to the server.

[1348] 3. The server uses the AI ​​to generate the optimal menu based on the user's health checkup data and nutritional balance information.

[1349] 4. The generation AI generates a menu that takes nutritional balance into consideration and sends it back to the server.

[1350] 5. The server sends the menu information to the terminal, and the terminal displays the menu to the user.

[1351] For example, a menu might include "Breakfast: oatmeal and fruit, Lunch: grilled salmon and salad, Dinner: chicken and vegetable soup."

[1352] In this way, a cooking assistance system utilizing interactive generative artificial intelligence can provide users with customized recipes and procedures, professional techniques, and healthy menus, providing practical and effective support.

[1353] The processing flow will be explained below.

[1354] Collection of User Information

[1355] Step 1:

[1356] The user launches the app for the first time and is presented with a screen to enter user information.

[1357] Step 2:

[1358] The device will present the user with fields to enter their name, age, food preferences, allergy information, and health information.

[1359] Step 3:

[1360] The user enters the required information and presses the send button.

[1361] Step 4:

[1362] The terminal transmits the input information to the server.

[1363] Step 5:

[1364] The server stores the received user information in a database.

[1365] Providing cooking recipes

[1366] Step 1:

[1367] The user voices or types in a request for a recipe using a specific ingredient (e.g., "Tell me some recipes using tomatoes").

[1368] Step 2:

[1369] The terminal transmits the input request to the server.

[1370] Step 3:

[1371] The server references the user's profile information from the database and sends the request to the generation artificial intelligence.

[1372] Step 4:

[1373] Generative AI generates the optimal recipe based on the user's profile and specified ingredients.

[1374] Step 5:

[1375] The server receives the generated recipe information and transmits it to the terminal.

[1376] Step 6:

[1377] The device displays the generated recipe (e.g., "Pasta with Tomato and Basil") to the user.

[1378] Real-time assistance with cooking procedures

[1379] Step 1:

[1380] If a user is cooking and wants to know the next step, they can send a request by voice or text (e.g., "Tell me the next step").

[1381] Step 2:

[1382] The device sends a request to the server.

[1383] Step 3:

[1384] The server issues instructions to the generation artificial intelligence to generate the next step.

[1385] Step 4:

[1386] Generative AI generates next step information (e.g., "Please finely chop the tomatoes").

[1387] Step 5:

[1388] The server receives the generated procedure information and transmits it to the terminal.

[1389] Step 6:

[1390] The device will provide the user with next steps via voice or text.

[1391] Step 7:

[1392] The user follows the instructions and sends the request again if the next step is required.

[1393] Providing professional technical information

[1394] Step 1:

[1395] A user submits a request for pro technical information on a recipe (e.g., "What are some pro tips for this recipe?").

[1396] Step 2:

[1397] The device sends a request to the server.

[1398] Step 3:

[1399] The server queries a professional technical information database and provides the information to the generating artificial intelligence.

[1400] Step 4:

[1401] Generative AI optimizes technical information and generates specific advice (e.g., "Professionals recommend blanching tomatoes before chopping them to make them taste even better").

[1402] Step 5:

[1403] The server receives the optimized technical information and sends it to the terminal.

[1404] Step 6:

[1405] The terminal displays or audibly explains technical information to the user.

[1406] Healthy menu suggestions

[1407] Step 1:

[1408] If a user wants to know about healthy meals, they send a request (e.g., "Tell me about healthy meals").

[1409] Step 2:

[1410] The device sends a request to the server.

[1411] Step 3:

[1412] The server retrieves the user's health information and nutritional balance information from the database and provides it to the generating artificial intelligence.

[1413] Step 4:

[1414] Generative artificial intelligence generates nutritionally balanced menus based on health information (e.g., "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup").

[1415] Step 5:

[1416] The server receives the generated menu information and transmits it to the terminal.

[1417] Step 6:

[1418] The terminal displays the menu information to the user.

[1419] The above are the specific processing steps of the cooking assistance system that uses interactive generative artificial intelligence.

[1420] Example 1

[1421] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1422] Current cooking assistance systems often fail to adequately address the individual needs and circumstances of users. Specifically, they lack customized recipes and assistance that reflect the user's health status, food preferences, allergy information, etc. They also lack real-time assistance needed during cooking and a means to easily obtain professional technical information. Furthermore, there is a lack of healthy menu suggestions based on the user's health information, so more personalized assistance is needed.

[1423] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1424] In this invention, the server includes means for inputting user information and transmitting it to the server, means for saving the input user information, means for having the AI ​​generate a recipe based on specified ingredients and providing the recipe to the user, means for providing the user with the next steps in real time by voice or text when the user actually starts cooking and requests the next steps, means for obtaining professional technical information for a specific recipe, optimizing it, and providing it to the user, and means for having the AI ​​generate a healthy menu based on health-related information and providing the menu to the user. This makes it possible to provide customized cooking support and real-time support that meets the individual needs of the user, as well as professional technical information and healthy menu suggestions.

[1425] "User information" is a general term for data related to individual needs and circumstances, such as a user's name, age, food preferences, allergy information, and health information.

[1426] A "server" is a computer system that receives and stores information sent by a user, sends requests to the generating artificial intelligence, processes the results, and provides them to the user.

[1427] A "terminal" is a device through which a user inputs information, makes requests, and receives and displays responses from a server.

[1428] "Generative AI" is an AI model that generates optimal recipes, procedures, technical information, and menus based on a user's requests and profile.

[1429] A "recipe" is a list of instructions or ingredients for cooking a particular dish.

[1430] "Real-time assistance" refers to assistance that provides the user with the next cooking steps on the spot as they proceed with their cooking.

[1431] "Professional technical information" is information about the techniques and tips of professional chefs.

[1432] A "healthy menu" refers to a meal plan that takes into consideration the user's health and nutritional balance.

[1433] The present invention is a cooking assistance system that uses interactive generative artificial intelligence to provide users with customized cooking recipes, cooking procedures, professional technical information, and healthy menus. Specific embodiments of the present invention are described below.

[1434] The system mainly consists of a server, a terminal, and a user. The server collects user information, stores and processes data, and runs generative AI models. The terminal receives input from the user, communicates with the server, and displays information to the user.

[1435] Hardware and software used

[1436] Hardware:

[1437] Server: A high-performance computer (e.g., a cloud service server)

[1438] Devices: smartphones, tablets, computers

[1439] software:

[1440] Generative AI models: such as OpenAI's GPT-3

[1441] Database: MySQL, MongoDB

[1442] Communication method: REST API, WebSocket

[1443] Front-end technologies: HTML, JavaScript, Swift

[1444] Collection of User Information

[1445] When the user first starts the app, they enter their name, age, food preferences, allergy information, and health information on the input screen on their device. The entered information is sent from the device to the server and stored in a database on the server. This creates a foundation for providing support customized for each user.

[1446] Providing cooking recipes

[1447] If a user wants to know about recipes that use a specific ingredient, they can make a request, for example, "Tell me recipes that use tomatoes." This request is sent from the device to the server, which then uses the user's profile information and the request to generate the optimal recipe for the generative AI model. The generative AI model then sends the generated recipe information back to the server, which then sends it to the device and displays it to the user.

[1448] Example: "Tomato and basil pasta"

[1449] Real-time assistance with cooking procedures

[1450] When a user actually starts cooking and wants to know the next step, they can make a request, for example, "Tell me the next step." This request is sent from the device to the server, and the server has the generative AI model generate the next step and send the result back to the server. The server then sends the step information to the device, and the device provides the step to the user via voice or text.

[1451] Example: "Please chop the tomatoes finely."

[1452] Providing professional technical information

[1453] If a user wants to know professional technical information for a particular recipe, they can make a request, for example, "Tell me the professional tips for this recipe." This request is sent from the device to the server, which queries a database of professional technical information and optimizes that information for the generative AI model. The optimized technical information is then sent from the server to the device, which then provides that information to the user.

[1454] Example: "Professionals blanch tomatoes before chopping them to enhance their flavor."

[1455] Healthy menu suggestions

[1456] If a user wants to know about healthy menus, they can make a request, for example, "Tell me about healthy menus." This request is sent from the device to the server, and the server uses the user's health checkup data and nutritional balance information to generate an optimal menu using the generative AI model. The generated menu is then sent to the device via the server and displayed to the user.

[1457] Example: "Breakfast: Oatmeal and fruit, Lunch: Grilled salmon and salad, Dinner: Chicken and vegetable soup."

[1458] As described above, the cooking assistance system of the present invention can provide users with customized recipes and procedures, professional technical information, and healthy menus, thereby providing practical and effective support.

[1459] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1460] Step 1:

[1461] Collection of User Information

[1462] Input: The user enters their name, age, food preferences, allergy information, and health information into the terminal.

[1463] Processing: The terminal sends the entered information to the server, which receives the information and processes it to save it in a database. Specifically, the data sent from the terminal undergoes validation checks on the server side before being saved in a database (for example, MySQL or MongoDB).

[1464] Output: Save to database, and user information is stored on the server.

[1465] Specific operation: When the user presses the send button, the message "Sent. Thank you." is displayed on the terminal.

[1466] Step 2:

[1467] Providing cooking recipes

[1468] Input: The user types a request into the terminal, such as "Tell me some recipes using tomatoes."

[1469] Processing: The device sends this request to the server, which then sends the received request and the user's profile information to the generative AI model (e.g., OpenAI's GPT-3). The generative AI model generates the optimal recipe based on the prompt, which includes the user's profile information (such as name, age, and food preferences).

[1470] Output: The recipe information generated by the generative AI model is sent back to the server, which then sends it to the device and displays it to the user.

[1471] Specific operation: An image of "Tomato and Basil Pasta" and recipe instructions will be displayed on the device screen.

[1472] Step 3:

[1473] Real-time assistance with cooking procedures

[1474] Input: The user types a request into the terminal, such as "Tell me what to do next."

[1475] Processing: The device sends a request to the server, which then sends it to the generative AI model to generate the next step. The prompt contains the current step and the next step. The generative AI model generates the next step and sends the result back to the server.

[1476] Output: The generated next step information is sent from the server to the terminal and provided to the user by voice or text.

[1477] Specific operation: The device will say, "Please finely chop the tomatoes."

[1478] Step 4:

[1479] Providing professional technical information

[1480] Input: A user types a request into a device, such as "Tell me some pro tips for this recipe."

[1481] Processing: The device sends a request to the server, which queries a professional technical information database to obtain information and optimizes it for the generative AI model. The prompt contains the recipe and related technical information. The generative AI model returns the optimized technical information to the server.

[1482] Output: The optimized technical information is sent from the server to the terminal and displayed or explained to the user by voice.

[1483] Specific action: The information displayed is, "Professionals blanch tomatoes before chopping them to make them taste even better."

[1484] Step 5:

[1485] Healthy menu suggestions

[1486] Input: The user inputs a request into the terminal, such as "Tell me some healthy meals."

[1487] Processing: The device sends a request to the server, which then sends a prompt to the generative AI model to generate an optimal menu based on the user's health checkup data and nutritional balance information. The generative AI model generates a menu that takes nutritional balance into consideration and returns the results to the server.

[1488] Output: The generated menu information is sent from the server to the terminal and displayed to the user.

[1489] Specific behavior: The menu will be displayed: "Breakfast: Oatmeal and fruit, Lunch: Grilled salmon and salad, Dinner: Chicken and vegetable soup."

[1490] (Application example 1)

[1491] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1492] Currently, there are a wide variety of systems that provide cooking instructions and recipes, but these systems generally do not take into account the user's individual preferences, health status, or the specific ingredients used. Even in brick-and-mortar stores, customers are rarely offered recipe suggestions based on the ingredients they have on hand, and it is difficult to obtain professional technical advice or next cooking steps in real time. As a result, users often have to go through a lot of effort to obtain the information they want, resulting in a lack of convenience.

[1493] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1494] In this invention, the server includes a means for inputting user information and transmitting it to the server, a means for saving the input user information, and a means for causing a recipe generation AI to generate a recipe based on designated ingredients and providing the recipe to the user. This makes it possible to propose recipes based on ingredients in-store using advanced information terminals. It also makes it possible to provide the user with next steps in real time by voice or text.

[1495] Furthermore, by providing a means to acquire professional technical information, optimize it, and provide it to users, it is possible to convey professional cooking techniques and tips to users. In addition, by using health-related information to generate healthy menus using artificial intelligence, it is possible to provide these menus to users, making it possible to suggest optimal menus based on each individual's health status. This will greatly improve user convenience and provide optimal cooking support.

[1496] "Interactive generative artificial intelligence" is an artificial intelligence that collects information through dialogue with users and generates and provides appropriate information and services based on that information.

[1497] A "cooking assistance system" is a system that assists users when cooking by suggesting recipes and guiding them through cooking procedures.

[1498] "User Information" refers to personal information such as the user's name, age, food preferences, allergy information, and health information.

[1499] A "server" is a computer system used to store user information, process data, and generate recipes.

[1500] "Advanced information terminals" refer to electronic devices with advanced functions, such as smartphones and digital kiosks.

[1501] A "recipe" is a collection of information on how to make a dish, such as the ingredients to be used, their amounts, and cooking steps.

[1502] "Professional technical information" refers to information related to cooking techniques and tips possessed by professional chefs.

[1503] "Health-related information" refers to data related to maintaining and improving health, such as a user's health checkup data and nutritional balance information.

[1504] A "menu" refers to a meal combination or menu plan for a specific period of time.

[1505] The present invention relates to a cooking assistance system that uses interactive generative artificial intelligence and aims to provide optimal assistance to users when cooking. Specific embodiments will be described below.

[1506] 1. System Configuration

[1507] This system consists of a terminal for inputting user information, a server for storing and processing data, and multiple functions that utilize the generative AI model. The terminals are advanced information terminals such as smartphones and digital kiosks. The server stores user information and processes the data, and generates various data using a generative AI model (e.g., OpenAI's GPT-4).

[1508] 2. Collection of User Information

[1509] The server collects user information (such as name, age, food preferences, allergy information, and health information) sent from the device and stores it in a database, enabling it to provide customized services tailored to each individual user.

[1510] 3. Recipe Generation

[1511] When a user wants to know a recipe based on specific ingredients, the device sends the request to the server. The server uses the user information and the request to generate the optimal recipe using a generative AI model, which then sends the recipe back to the device. For example, a recipe such as "pasta with tomatoes and basil" is generated and served to the user.

[1512] 4. Providing real-time procedures

[1513] When a user wants real-time instructions for cooking a dish, the device sends a request to the server, which uses a generative AI model to generate the next steps and sends them back to the device in real time, providing specific instructions via voice or text, such as "finely chop the tomatoes."

[1514] 5. Providing professional technical information

[1515] When a user wants to know professional technical information for a particular recipe, the device sends the request to the server. The server queries a database of professional technical information, optimizes the information with a generative AI model, and provides it to the user. For example, the information provided may be, "Professionals recommend blanching the tomatoes before chopping them to enhance the flavor."

[1516] 6. Healthy meal suggestions

[1517] When a user wants to know about healthy meals, the device sends a request to the server. The server uses the user's health checkup data and nutritional balance information to generate an optimal meal plan using the generative AI model, and sends the plan back to the device. For example, a menu such as "Breakfast: Oatmeal and fruit, Lunch: Grilled salmon and salad, Dinner: Chicken and vegetable soup" may be provided.

[1518] Hardware and Software

[1519] The server is built using programming languages ​​and frameworks such as Python and Flask, and uses OpenAI's API for generative AI, with user information stored in a data storage system such as an SQLite database.

[1520] Specific examples

[1521] Examples of prompts include:

[1522] "Tell me a recipe that uses tomatoes, basil, and pasta."

[1523] "Tell me the next cooking step"

[1524] "What are some pro tips for this recipe?"

[1525] By using these prompts, users can obtain optimal recipes, cooking procedures, and specialized technical information in real time. In this way, a cooking assistance system using interactive generative AI can provide users with highly convenient and effective support.

[1526] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1527] Step 1:

[1528] Collection of User Information

[1529] Users enter their name, age, food preferences, allergy information, and health information through a device (smartphone or digital kiosk). The entered data is sent from the device to a server, which receives the data and stores it in an SQLite database. This creates the foundation for providing services customized for each user.

[1530] Step 2:

[1531] Recipe Generation

[1532] When a user wants to know a recipe based on a specific ingredient, they can input, for example, "Tell me a recipe using tomatoes" into their device. The device then sends this request to the server. The server references the user information and the request and generates the optimal recipe based on a generative AI model (for example, GPT-4). The generated recipe is then sent back from the server to the device and displayed to the user. Specifically, in response to the prompt "Tell me a recipe using tomatoes, basil, and pasta," "Pasta with tomatoes and basil" is suggested.

[1533] Step 3:

[1534] Providing real-time procedures

[1535] When a user wants real-time support for cooking steps, they can type, for example, "Tell me the next steps" into their device. Upon receiving this request, the device sends a request to the server. The server generates the next cooking steps based on a generative AI model. These steps are sent back from the server to the device and provided to the user via text or voice. Specifically, instructions such as "Please finely chop the tomatoes" are provided in real time.

[1536] Step 4:

[1537] Providing professional technical information

[1538] If a user wants to know professional technical information for a particular recipe, they can type, for example, "Tell me the pro tips for this recipe" into their device. The device then sends this request to the server. The server queries a database of professional technical information and optimizes the information using a generative AI model. The optimized information is then sent back from the server to the device and presented to the user via display or audio. For example, they might provide information such as, "Professionals blanch tomatoes before chopping them to enhance their flavor."

[1539] Step 5:

[1540] Healthy menu suggestions

[1541] If a user wants to know what a healthy menu is, they can type, for example, "Tell me what a healthy menu is." The device sends this request to the server. The server uses the user's health checkup data and nutritional balance information to have the generative AI model generate the optimal menu. This menu information is sent back from the server to the device and provided to the user. Specifically, the suggested menu might be "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup."

[1542] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1543] The present invention is a cooking assistance system that utilizes an emotion engine in addition to interactive generative artificial intelligence, and provides more personalized assistance by recognizing the user's emotions and customizing recipes and procedures based on those emotions, and adjusting the tone and content of the dialogue. Specific embodiments of the present invention are described below.

[1544] System Overview

[1545] Users can interact with the system through an interface and receive various cooking assistance. The system utilizes information from the server, the device, and the user, and also uses an emotion engine to suggest recipes, provide real-time assistance with procedures, provide professional technical information, and suggest healthy meals.

[1546] Program processing overview

[1547] The system works through user information collection, recipe generation, real-time assistance, professional technical information provision, healthy meal suggestions, as well as emotion recognition and optimization using an emotion engine.

[1548] Collection of User Information

[1549] 1. The user launches the app for the first time and is prompted to enter user information.

[1550] 2. The device displays fields for the user to enter their name, age, food preferences, allergy information, and health information.

[1551] 3. The user enters this information and presses the submit button.

[1552] 4. The terminal sends the entered information to the server.

[1553] 5. The server stores the received user information in a database.

[1554] Providing cooking recipes

[1555] 1. A user voices or texts a request for a recipe using a specific ingredient (e.g., "Tell me some recipes using tomatoes").

[1556] 2. The terminal sends the input request to the server.

[1557] 3. The server uses the AI ​​to generate the optimal recipe based on the user's profile information and request.

[1558] 4. The generation AI returns the generated recipe information to the server.

[1559] 5. The server sends the recipe information to the terminal, and the terminal displays the generated recipe (e.g., "Pasta with Tomato and Basil") for the user.

[1560] Real-time assistance with cooking procedures

[1561] 1. If a user is cooking and wants to know the next step, they can send a request via voice or text (e.g., "Tell me the next step").

[1562] 2. The device sends a request to the server.

[1563] 3. The server has the generation AI generate the next steps and send those steps back to the server.

[1564] 4. The generative AI generates next step information (e.g., "Please finely chop the tomatoes").

[1565] 5. The server receives the generated procedure information and sends it to the terminal.

[1566] 6. The device will provide the user with next steps via voice or text.

[1567] Providing professional technical information

[1568] 1. A user submits a request for pro technical information on a recipe (e.g., "What are some pro tips for this recipe?").

[1569] 2. The device sends a request to the server.

[1570] 3. The server queries a professional technical information database and provides that information to the generation AI.

[1571] 4. Generative AI optimizes technical information and generates specific advice (e.g., "Professionals recommend blanching tomatoes before chopping them to make them taste even better").

[1572] 5. The server receives the optimized technical information and sends it to the device.

[1573] 6. The device displays or audibly explains technical information to the user.

[1574] Healthy menu suggestions

[1575] 1. If a user wants to know about healthy meals, they send a request (e.g., "Tell me about healthy meals").

[1576] 2. The device sends a request to the server.

[1577] 3. The server retrieves the user's health checkup data and nutritional balance information from the database and provides it to the generation AI.

[1578] 4. Generative AI generates nutritionally balanced menus based on health information (e.g., "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup").

[1579] 5. The server receives the generated menu information and sends it to the terminal.

[1580] 6. The device displays the menu information to the user.

[1581] Utilizing the Emotion Engine

[1582] 1. If the user's emotions change during a conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion.

[1583] 2. The device sends the recognized emotion data to the server.

[1584] 3. The server provides the generative AI with emotional data and issues instructions to customize the recipe, steps, and dialogue content.

[1585] For example, if a user is feeling stressed about cooking, the emotion engine will recognize this and the generative AI will suggest recipes that are easy and quick to make. This allows users to receive optimal cooking assistance tailored to their mood and situation.

[1586] In this way, a cooking assistance system that combines interactive generative artificial intelligence and an emotion engine can provide users with customized recipes and procedures, professional techniques, and healthy menus, and can also respond to their emotions, providing more personalized assistance.

[1587] The processing flow will be explained below.

[1588] Processing flow of a cooking support system using an emotion engine

[1589] Collection of User Information

[1590] Step 1:

[1591] The user launches the app for the first time and is presented with a screen to enter user information.

[1592] Step 2:

[1593] The device will present the user with fields to enter their name, age, food preferences, allergy information, and health information.

[1594] Step 3:

[1595] The user enters this information and presses the send button.

[1596] Step 4:

[1597] The terminal transmits the input information to the server.

[1598] Step 5:

[1599] The server stores the received user information in a database.

[1600] Providing cooking recipes

[1601] Step 1:

[1602] The user voices or types in a request for a recipe using a specific ingredient (e.g., "Tell me some recipes using tomatoes").

[1603] Step 2:

[1604] The terminal transmits the input request to the server.

[1605] Step 3:

[1606] The server references the user's profile information from the database and sends the request to the generation artificial intelligence.

[1607] Step 4:

[1608] Generative AI generates the optimal recipe based on the user's profile and specified ingredients.

[1609] Step 5:

[1610] The server receives the generated recipe information and transmits it to the terminal.

[1611] Step 6:

[1612] The device displays the generated recipe (e.g., "Pasta with Tomato and Basil") to the user.

[1613] Real-time assistance with cooking procedures

[1614] Step 1:

[1615] If a user is cooking and wants to know the next step, they can send a request by voice or text (e.g., "Tell me the next step").

[1616] Step 2:

[1617] The device sends a request to the server.

[1618] Step 3:

[1619] The server issues instructions to the generation artificial intelligence to generate the next step.

[1620] Step 4:

[1621] Generative AI generates next step information (e.g., "Please finely chop the tomatoes").

[1622] Step 5:

[1623] The server receives the generated procedure information and transmits it to the terminal.

[1624] Step 6:

[1625] The device will provide the user with next steps via voice or text.

[1626] Providing professional technical information

[1627] Step 1:

[1628] A user submits a request for pro technical information on a recipe (e.g., "What are some pro tips for this recipe?").

[1629] Step 2:

[1630] The device sends a request to the server.

[1631] Step 3:

[1632] The server queries a professional technical information database and provides the information to the generating artificial intelligence.

[1633] Step 4:

[1634] Generative AI optimizes technical information and generates specific advice (e.g., "Professionals recommend blanching tomatoes before chopping them to make them taste even better").

[1635] Step 5:

[1636] The server receives the optimized technical information and sends it to the terminal.

[1637] Step 6:

[1638] The terminal displays or audibly explains technical information to the user.

[1639] Healthy menu suggestions

[1640] Step 1:

[1641] If a user wants to know about healthy meals, they send a request (e.g., "Tell me about healthy meals").

[1642] Step 2:

[1643] The device sends a request to the server.

[1644] Step 3:

[1645] The server retrieves the user's health checkup data and nutritional balance information from the database and provides it to the generating artificial intelligence.

[1646] Step 4:

[1647] Generative artificial intelligence generates nutritionally balanced menus based on health information (e.g., "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup").

[1648] Step 5:

[1649] The server receives the generated menu information and transmits it to the terminal.

[1650] Step 6:

[1651] The terminal displays the menu information to the user.

[1652] Utilizing the Emotion Engine

[1653] Step 1:

[1654] If the user's emotions change during the conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion.

[1655] Step 2:

[1656] The device transmits the recognized emotion data to the server.

[1657] Step 3:

[1658] The server provides emotional data to the generative AI, which then issues instructions to customize recipes, steps, and dialogue content.

[1659] As a concrete example, if a user is feeling stressed about cooking, the emotion engine will recognize this and the generative AI will suggest recipes that are easy and quick to make. In this example, it is possible to suggest "time-saving recipes that can be made in the microwave." This allows users to receive optimal cooking support tailored to their mood and situation.

[1660] Example 2

[1661] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1662] In today's modern lifestyles, it is extremely important to provide healthy and efficient cooking assistance that accommodates the different lifestyles and food preferences of each individual user. However, conventional cooking assistance systems lack the ability to customize based on the user's emotions and health status, making it difficult to provide more personalized assistance. There is also a need to effectively incorporate technical information from experts to improve the quality of cooking. Furthermore, in situations where real-time response is required, providing prompt and appropriate information has been insufficient.

[1663] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1664] In this invention, the server includes means for inputting user information and transmitting it to the server, means for saving the input user information, means for having a generation AI generate a recipe based on specified ingredients and providing the recipe to the user, means for providing the user with real-time audio or text instructions for the next steps, means for acquiring technical information from experts and optimizing it to provide to the user, means for having the generation AI generate a healthy menu based on health-related information and providing the menu to the user, and means for analyzing the user's emotions using an emotion recognition engine and customizing the recipe, steps, and dialogue content based on the emotions. This enables personalized cooking support according to the individual needs and emotions of the user, provision of menus that accommodate health management, and high-quality cooking support utilizing the skills of experts.

[1665] "User information" refers to the name, age, food preferences, allergy information, health information, etc. entered by the user.

[1666] "Server" refers to a computer system that stores input user information, processes various data, and provides it to the generation AI.

[1667] "Specified ingredients" refers to ingredients that the user specifies to be used in creating the recipe.

[1668] "Generative AI" refers to an AI model that generates recipes, steps, menus, etc. based on user requests.

[1669] "Real-time" refers to nearly immediate response to user requests.

[1670] "Expert technical information" refers to information about professional knowledge and techniques related to cooking.

[1671] "Health-related information" refers to all data related to health and nutrition, such as a user's health checkup data and nutritional balance information.

[1672] An "emotion recognition engine" refers to a system that recognizes emotions by analyzing the user's voice and facial expressions.

[1673] "Personalized cooking assistance" refers to providing cooking assistance customized based on the user's individual information and emotions.

[1674] "Customizing dialogue content" refers to adjusting the tone and content of dialogue provided according to the user's emotions and situation.

[1675] The present invention is a cooking assistance system that utilizes interactive generative artificial intelligence (generative AI model) and an emotion engine to recognize the user's emotions and customize recipes and procedures according to those emotions, thereby providing more personalized assistance. A specific embodiment of this system is shown below.

[1676] Overall system overview

[1677] Users can interact with the system through devices such as smartphones and tablets and receive various cooking assistance. The system utilizes information from the server, devices, and users, and uses an emotion engine to suggest recipes, provide real-time assistance with procedures, provide expert technical information, and suggest healthy meals. The system mainly uses the following hardware and software:

[1678] Hardware: smartphones, tablets, servers, database servers

[1679] Software: Interactive generative AI models, emotion recognition engines, user interface applications, database management systems

[1680] Program processing overview

[1681] The system works through user information collection, recipe generation, real-time assistance, expert technical input, healthy meal suggestions, as well as emotion recognition and optimization using an emotion engine.

[1682] Collection of User Information

[1683] When a user launches the app for the first time, a screen for entering user information is displayed. The device displays fields for entering the user's name, age, food preferences, allergy information, and health information. When the user enters this information and presses the send button, the device sends the entered information to the server. The server stores the received user information in a database.

[1684] Providing cooking recipes

[1685] A user inputs a request for a recipe using a specific ingredient via voice or text. For example, if the request is "Tell me a recipe using tomatoes," the device analyzes the input request and sends it to the server. The server uses the user's profile information and the request content to have the generative AI model generate the optimal recipe. The generative AI generates recipe information, such as "pasta with tomatoes and basil," and sends it back to the server. The server then sends the generated recipe information to the device and displays it to the user.

[1686] Real-time assistance with cooking procedures

[1687] When a user wants to know the next step while cooking, they send a request by voice or text. For example, if a request is made to "tell me the next step," the device sends the request to the server. The server then has the generative AI model generate the next step and sends that step back to the server. The generative AI generates step information such as "finely chop the tomatoes," and the server sends that information to the device. The device then provides the next step to the user by voice or text.

[1688] Providing technical information from experts

[1689] A user sends a request for expert technical information about a recipe. For example, if the user requests, "Tell me the pro tips for this recipe," the device sends the request to the server. The server queries a database of expert technical information and provides that information to the generative AI model. The generative AI generates specific technical information, such as "Professionals recommend blanching tomatoes before chopping them to enhance their flavor," and sends it back to the server. The server then sends that information to the device and explains it to the user by displaying it or by voice.

[1690] Healthy menu suggestions

[1691] When a user wants to know about healthy menu options, they send a request. For example, if the request is "Tell me about healthy menu options," the device sends the request to the server. The server retrieves the user's health checkup data and nutritional balance information from a database and provides this to the generative AI model. The generative AI generates a healthy menu option such as "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup." The server sends the generated menu information to the device and displays it to the user.

[1692] Utilizing the Emotion Engine

[1693] If the user's emotions change during a conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion. The device then sends the recognized emotion data to the server, which then provides the emotion data to the generative AI model and issues instructions to customize the recipe, steps, and conversation content. For example, if the user is feeling stressed about cooking, the emotion engine will recognize this, and the generative AI model will suggest a recipe that is easy and quick to make. This allows the user to receive optimal cooking assistance tailored to their mood and situation.

[1694] Examples of prompt statements

[1695] Recipe generation prompt: "Generate a quick recipe using tomatoes. The user is stressed."

[1696] Technical information prompt: "What are some pro tips for this recipe?"

[1697] In this way, the cooking assistance system of the present invention combines interactive generative artificial intelligence with an emotion engine to provide users with customized recipes, procedures, expert techniques, and healthy menus, and can also respond to emotions, making it possible to provide more personalized assistance.

[1698] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1699] Collection of User Information

[1700] Step 1:

[1701] The user launches the app for the first time and is presented with a screen to enter user information.

[1702] What it does: The device displays fields for entering name, age, food preferences, allergy information, and health information.

[1703] Step 2:

[1704] The user enters this information and presses the send button.

[1705] Specific operation: The device receives the entered information, detects when the send button is pressed, and sends it to the server.

[1706] Input: User's personal information (name, age, food preferences, allergy information, health information)

[1707] Output: User information sent to the server

[1708] Step 3:

[1709] The terminal transmits the input information to the server.

[1710] Specific operation: The terminal sends the received user information to the server via a secure communication channel.

[1711] Input: User information (secure format)

[1712] Output: User information sent to the server

[1713] Step 4:

[1714] The server stores the received user information in a database.

[1715] Specific operation: The server stores the received user information in a database and makes it available for subsequent processes.

[1716] Input: User information sent to the server

[1717] Output: User information stored in the database

[1718] Providing cooking recipes

[1719] Step 1:

[1720] A user voices or texts a request for a recipe using a specific ingredient.

[1721] Specific action: For example, request "Tell me some recipes that use tomatoes."

[1722] Input: Request for a specific material

[1723] Output: The text or audio data of the request

[1724] Step 2:

[1725] The terminal transmits the input request to the server.

[1726] Specific operation: The device performs voice recognition and text analysis and sends the request content to the server.

[1727] Input: The text or audio data of the request

[1728] Output: The request sent to the server

[1729] Step 3:

[1730] The server generates a prompt for the generative AI model based on the user's profile information and request content.

[1731] Specific operation: The server converts the user information and request content into a prompt text.

[1732] Input: User profile information and request details

[1733] Output: Prompt sentence for generative AI model

[1734] Step 4:

[1735] The generative AI model generates recipe information based on the prompt sentence and sends it back to the server.

[1736] Specific behavior: The generative AI generates specific recipes such as "pasta with tomato and basil."

[1737] Input: prompt statement

[1738] Output: Generated recipe information

[1739] Step 5:

[1740] The server transmits the generated recipe information to the terminal.

[1741] Specific operation: The server transfers the recipe information to the device.

[1742] Input: Generated recipe information

[1743] Output: Recipe information sent to the device

[1744] Step 6:

[1745] The terminal displays the generated recipe to the user.

[1746] Specific operation: The device displays the received recipe information on the user interface.

[1747] Input: Recipe information sent to the device

[1748] Output: Recipe information displayed to the user

[1749] Real-time assistance with cooking procedures

[1750] Step 1:

[1751] If a user is cooking and wants to know the next step, they can send a request by voice or text.

[1752] Specific behavior: For example, request "Tell me what the next step is."

[1753] Input: Next Step Request

[1754] Output: The text or audio data of the request

[1755] Step 2:

[1756] The device sends a request to the server.

[1757] Specific operation: The device performs voice recognition or text analysis and forwards the request content to the server.

[1758] Input: The text or audio data of the request

[1759] Output: The request sent to the server

[1760] Step 3:

[1761] The server asks the generative AI model to generate the next step as a prompt.

[1762] Specific operation: The server summarizes the current procedure information into a prompt sentence and passes it to the generative AI model.

[1763] Input: Current procedure information

[1764] Output: Prompt sentence for generative AI model

[1765] Step 4:

[1766] The generative AI model generates next step information and sends it back to the server.

[1767] Specific actions: The generative AI generates specific steps such as "finely chop the tomatoes."

[1768] Input: prompt statement

[1769] Output: Generated procedure information

[1770] Step 5:

[1771] The server transmits the generated procedure information to the terminal.

[1772] Specific operation: The server transfers the generated procedure information to the terminal.

[1773] Input: Generated procedure information

[1774] Output: Instructions sent to the terminal

[1775] Step 6:

[1776] The device will provide the user with next steps via voice or text.

[1777] Specific operation: The device explains the received procedure information to the user by voice or text.

[1778] Input: Instructions sent to the terminal

[1779] Output: Next steps information provided to the user

[1780] Providing technical information from experts

[1781] Step 1:

[1782] A user submits a request for expert technical information on a recipe.

[1783] Specific behavior: For example, request "Tell me some pro tips for this recipe."

[1784] Input: Technical Information Request

[1785] Output: The text or audio data of the request

[1786] Step 2:

[1787] The device sends a request to the server.

[1788] Specific operation: The device performs voice recognition or text analysis and forwards the request content to the server.

[1789] Input: The text or audio data of the request

[1790] Output: The request sent to the server

[1791] Step 3:

[1792] The server queries a database of technical information from experts.

[1793] Specific operation: The server refers to the database based on the request and retrieves technical information.

[1794] Input: Request details

[1795] Output: Expert technical information

[1796] Step 4:

[1797] The generative AI model generates specific, optimized advice based on technical information and sends it back to the server.

[1798] Specific operation: The generative AI generates advice such as, "Professionals recommend blanching tomatoes before chopping them to make them taste even better."

[1799] Input: Expert technical information

[1800] Output: The generated advice

[1801] Step 5:

[1802] The server transmits the generated advice to the terminal.

[1803] Specific operation: The server transfers the generated advice to the terminal.

[1804] Input: Generated advice

[1805] Output: Advice sent to terminal

[1806] Step 6:

[1807] The terminal displays or audibly explains the advice to the user.

[1808] Specific operation: The device displays the received advice on the user interface or explains it aloud.

[1809] Input: Advice sent to terminal

[1810] Output: Advice given to the user

[1811] Healthy menu suggestions

[1812] Step 1:

[1813] If a user wants to know about healthy meals, they send a request.

[1814] Specific action: For example, request "Tell me some healthy meals."

[1815] Input: Healthy Meal Request

[1816] Output: The text or audio data of the request

[1817] Step 2:

[1818] The device sends a request to the server.

[1819] Specific operation: The device performs voice recognition or text analysis and forwards the request content to the server.

[1820] Input: The text or audio data of the request

[1821] Output: The request sent to the server

[1822] Step 3:

[1823] The server retrieves the user's health checkup data and nutritional balance information from the database.

[1824] Specific operation: The server retrieves data related to the user's health from the database.

[1825] Input: User's health checkup data, nutritional balance information

[1826] Output: Health data collected by the server

[1827] Step 4:

[1828] The generative AI model generates menus that take nutritional balance into consideration based on health information.

[1829] Specific operation: The generative AI generates menus such as "Breakfast: oatmeal and fruit, Lunch: grilled salmon and salad, Dinner: chicken and vegetable soup."

[1830] Input: User's health checkup data and nutritional balance information

[1831] Output: Generated healthy meal plan

[1832] Step 5:

[1833] The server transmits the generated menu information to the terminal.

[1834] Specific operation: The server transfers the generated menu information to the terminal.

[1835] Input: Generated menu information

[1836] Output: Menu information sent to the device

[1837] Step 6:

[1838] The terminal displays the menu information to the user.

[1839] Specific operation: The device displays the received menu information on the user interface.

[1840] Input: Menu information sent to the device

[1841] Output: Menu information provided to the user

[1842] Utilizing the Emotion Engine

[1843] Step 1:

[1844] If the user's emotions change during the conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion.

[1845] How it works: The emotion engine analyzes the user's voice and facial expressions in real time to detect emotions such as stress or joy.

[1846] Input: User voice and facial expression data

[1847] Output: Recognized emotion data

[1848] Step 2:

[1849] The device transmits the recognized emotion data to the server.

[1850] Specific operation: The device sends emotion data to the server.

[1851] Input: Recognized emotion data

[1852] Output: Emotion data sent to the server

[1853] Step 3:

[1854] The server provides emotional data to a generative AI model, which then provides instructions to customize recipes, steps, and dialogue.

[1855] Specific operation: The server incorporates emotional data into a prompt sentence and passes it to the generative AI model.

[1856] Input: Emotion data

[1857] Output: Prompt sentence for generative AI model

[1858] Step 4:

[1859] The generative AI model generates emotion-based recipes and instructions and sends them back to the server.

[1860] For example, if a user is feeling stressed about cooking, the generative AI will generate a recipe such as, "If you're feeling stressed, make tomato and basil pasta. It's easy to make."

[1861] Input: prompt statement

[1862] Output: The generated recipe or instructions

[1863] Step 5:

[1864] The server sends the generated recipes and instructions to the device.

[1865] Specific operation: The server transfers the generated information to the terminal.

[1866] Input: Generated recipes and instructions

[1867] Output: Information sent to the terminal

[1868] Step 6:

[1869] The device displays or audibly explains the recipe and steps to the user.

[1870] Specific operation: The device displays or audibly explains the received information.

[1871] Input: Information sent to the device

[1872] Output: The recipe or instructions provided to the user

[1873] (Application example 2)

[1874] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1875] Conventional cooking assistance systems and production process management systems did not provide support that took into account the emotions of users and workers. As a result, they were unable to respond appropriately to situations that caused stress to users and workers, leading to problems such as reduced work efficiency and decreased motivation. In addition, it was difficult to respond to the individual needs of users and workers, resulting in a lack of personalized support.

[1876] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting user information and transmitting it to the server, means for saving the input user information, means for having a generation AI generate a recipe based on specified ingredients and providing the recipe to the user, means for providing the user with real-time audio or text instructions on how to proceed, means for acquiring professional technical information, optimizing it, and providing it to the user, means for collecting emotion data using an emotion engine that recognizes the user's emotions and customizing the content of the provision based on that emotion data, means for having a generation AI generate optimal support information based on the emotion data and work content of factory workers and providing it to the workers, and means for having a generation AI generate healthy menus based on health-related information and providing the menus to the user. This provides personalized support that takes the emotions of users and workers into consideration, thereby reducing stress and improving work efficiency.

[1877] "Interactive generative artificial intelligence" is an artificial intelligence that collects information through dialogue with the user and makes appropriate suggestions and assistance based on that information.

[1878] A "system" is a collection of devices and software in which multiple elements work together to achieve a specific function.

[1879] "User Information" means certain data about you, such as your name, age, food preferences, allergy information, and health information.

[1880] A "server" is a computer system that provides services and data to other computers on a network.

[1881] An "emotion engine" is software that analyzes the user's voice and facial expressions to recognize emotions and process information accordingly.

[1882] A "recipe" is a list of steps and ingredients for making a particular dish.

[1883] "Professional technical information" refers to specific techniques and know-how provided by experts.

[1884] A "healthy menu" is a combination of nutritionally balanced meals.

[1885] A "factory work support system" is a system that aims to improve the efficiency of work within a factory and increase quality and productivity.

[1886] "Workers" are workers who work in factories and other places.

[1887] "Emotion data" refers to information about the emotions of users and workers obtained by the emotion engine.

[1888] This invention is a factory work support system that uses interactive generative artificial intelligence and an emotion engine, and provides personalized support according to the emotions of users and workers.

[1889] System Overview

[1890] This system includes a terminal, a server, an emotion engine, and a generative AI model. The terminal provides an interface for users and workers to input information, and the server stores and processes the input information and generates appropriate support information.

[1891] Hardware and Software

[1892] 1. Hardware:

[1893] Cameras for facial recognition (e.g., high-performance webcams)

[1894] Microphone for voice analysis (e.g. high-sensitivity microphone)

[1895] Terminals for workers (e.g., tablet terminals)

[1896] 2. Software:

[1897] Emotion recognition libraries (e.g., emotion analysis APIs)

[1898] Generative AI models (e.g., interactive generative AI engines)

[1899] Database (e.g. SQL database)

[1900] Front-end (e.g., web browser-based interface)

[1901] Data processing and calculation

[1902] The server receives the user information and worker information sent from the terminal and performs the following data processing and calculations.

[1903] 1. Emotion recognition:

[1904] Cameras and microphones are used to collect the facial expressions and voices of workers.

[1905] Using the sentiment analysis API, the collected data is analyzed to obtain sentiment data.

[1906] 2. Generating support information:

[1907] Based on emotion data and task data, the generative AI model generates appropriate work procedures and support information.

[1908] The generated information is stored in a database and transmitted to the terminal as needed.

[1909] Specific use cases

[1910] Let's say a worker is performing assembly work in a factory. While working, a camera and microphone analyze the worker's facial expressions and voice, and an emotion engine detects stress. The server receives the stress data and sends a prompt to the generation AI saying, "The worker is in a stressful state, so please suggest a simplified procedure for the work." The generation AI generates a simplified procedure, and the server sends that information to the terminal, presenting the simple procedure to the worker.

[1911] Prompt Sentence Examples

[1912] "High Stress" state. Please simplify the current task and suggest next steps.

[1913] In this way, a factory work support system that combines interactive generative artificial intelligence and an emotion engine can provide flexible support that responds to the emotions of workers, improving work efficiency and worker satisfaction.

[1914] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1915] Step 1:

[1916] The terminal provides an interface for inputting user information. The user or worker inputs user information such as name, age, food preferences, allergy information, and health information, and presses the send button. The input data is sent to the server.

[1917] Step 2:

[1918] The server stores the received user information in a database, including user and worker profile information.

[1919] Step 3:

[1920] A request for cooking or task assistance is entered via voice or text from the device. For example, a request might be "Tell me a recipe using tomatoes" or "Tell me the next step in the task." The request is then sent to the server.

[1921] Step 4:

[1922] The server sends a prompt to the generative AI model based on the request content and user information to generate the optimal recipe and work procedure. An example of a prompt sentence is, "The worker is under stress, so please suggest a procedure that simplifies the work."

[1923] Step 5:

[1924] A generative AI model generates recipes and instructions based on the prompt, which are then sent back to the server. Examples include "pasta with tomatoes and basil" and instructions such as "finely chop the tomatoes."

[1925] Step 6:

[1926] The server sends the information received from the generation AI to the terminal, which then displays or speaks the generated recipes and work procedures to the user or worker.

[1927] Step 7:

[1928] The device's camera and microphone analyze the facial expressions and voices of users and workers in real time. The emotion engine recognizes emotions based on the analyzed data and sends specific emotional data to the server.

[1929] Step 8:

[1930] Based on the emotion data received by the server, the generative AI model is prompted again to generate assistance according to the emotion, resulting in the generation of customized recipes and work procedures according to the emotion.

[1931] Step 9:

[1932] The server sends the regenerated information to the terminal, which then presents the user or worker with optimized recipes and work procedures, reducing stress and enabling more efficient work.

[1933] The above processing steps realize a factory work support system that utilizes interactive generative AI and an emotion engine. The collaboration between the generative AI model and the emotion engine makes it possible to provide personalized support according to the emotions of users and workers.

[1934] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1935] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1936] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1937] [Fourth embodiment]

[1938] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1939] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1940] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1941] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1942] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1943] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1944] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1945] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1946] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1947] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1948] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1949] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1950] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1951] The present invention provides a cooking assistance system that utilizes interactive generative artificial intelligence. Specific embodiments of the system will be described below.

[1952] System Overview

[1953] Users can interact with the system through an interface and receive various cooking assistance. The system primarily utilizes information collected from the server, terminals, and users to suggest recipes, provide real-time assistance with procedures, provide professional technical information, and suggest health-based menus.

[1954] Program processing overview

[1955] The system operates through the following steps: collecting user information, generating recipes, providing real-time assistance, providing professional technical information, and suggesting healthy meals.

[1956] Collection of User Information

[1957] 1. When a user uses the system for the first time, the terminal displays a screen for the user to enter their name, age, food preferences, allergy information, and health information.

[1958] 2. The user enters this information and presses the submit button.

[1959] 3. The device sends the entered information to the server.

[1960] 4. The server stores the received user information in a database.

[1961] 5. This allows the system to build a foundation for providing customized assistance to each user.

[1962] Providing cooking recipes

[1963] 1. If a user wants to know recipes that use a specific ingredient, they can request, for example, "Tell me recipes that use tomatoes."

[1964] 2. The device receives this request and sends it to the server.

[1965] 3. The server uses the AI ​​to generate the optimal recipe based on the user's profile information and request.

[1966] 4. The generation AI returns the generated recipe information to the server.

[1967] 5. The server sends the recipe information to the terminal, and the terminal displays the recipe to the user.

[1968] As a specific example, a recipe such as "tomato and basil pasta" is generated and provided to the user.

[1969] Real-time assistance with cooking procedures

[1970] 1. When the user actually starts cooking and wants to know the next steps, they can request, for example, "Tell me the next steps."

[1971] 2. The device sends this request to the server.

[1972] 3. The server has the generation AI generate the next steps and send those steps back to the server.

[1973] 4. The server sends the procedure information to the terminal, and the terminal provides the procedure to the user by voice or text.

[1974] For example, instructions such as "Please finely chop the tomatoes" are provided in real time.

[1975] Providing professional technical information

[1976] 1. If a user wants to know professional technical information about a particular recipe, they can request, for example, "Tell me the pro tips for this recipe."

[1977] 2. The device sends this request to the server.

[1978] 3. The server queries a professional technical information database and optimizes the information for the generating AI.

[1979] 4. The generating AI provides optimized technical information to the server.

[1980] 5. The server sends the technical information to the terminal, and the terminal displays or audibly explains the technical information to the user.

[1981] As a specific example, information such as "Professionals blanch tomatoes before chopping them to make them taste even better" is provided.

[1982] Healthy menu suggestions

[1983] 1. If a user wants to know about healthy meals, they can request, for example, "Tell me about healthy meals."

[1984] 2. The device sends this request to the server.

[1985] 3. The server uses the AI ​​to generate the optimal menu based on the user's health checkup data and nutritional balance information.

[1986] 4. The generation AI generates a menu that takes nutritional balance into consideration and sends it back to the server.

[1987] 5. The server sends the menu information to the terminal, and the terminal displays the menu to the user.

[1988] For example, a menu might include "Breakfast: oatmeal and fruit, Lunch: grilled salmon and salad, Dinner: chicken and vegetable soup."

[1989] In this way, a cooking assistance system utilizing interactive generative artificial intelligence can provide users with customized recipes and procedures, professional techniques, and healthy menus, providing practical and effective support.

[1990] The processing flow will be explained below.

[1991] Collection of User Information

[1992] Step 1:

[1993] The user launches the app for the first time and is presented with a screen to enter user information.

[1994] Step 2:

[1995] The device will present the user with fields to enter their name, age, food preferences, allergy information, and health information.

[1996] Step 3:

[1997] The user enters the required information and presses the send button.

[1998] Step 4:

[1999] The terminal transmits the input information to the server.

[2000] Step 5:

[2001] The server stores the received user information in a database.

[2002] Providing cooking recipes

[2003] Step 1:

[2004] The user voices or types in a request for a recipe using a specific ingredient (e.g., "Tell me some recipes using tomatoes").

[2005] Step 2:

[2006] The terminal transmits the input request to the server.

[2007] Step 3:

[2008] The server references the user's profile information from the database and sends the request to the generation artificial intelligence.

[2009] Step 4:

[2010] Generative AI generates the optimal recipe based on the user's profile and specified ingredients.

[2011] Step 5:

[2012] The server receives the generated recipe information and transmits it to the terminal.

[2013] Step 6:

[2014] The device displays the generated recipe (e.g., "Pasta with Tomato and Basil") to the user.

[2015] Real-time assistance with cooking procedures

[2016] Step 1:

[2017] If a user is cooking and wants to know the next step, they can send a request by voice or text (e.g., "Tell me the next step").

[2018] Step 2:

[2019] The device sends a request to the server.

[2020] Step 3:

[2021] The server issues instructions to the generation artificial intelligence to generate the next step.

[2022] Step 4:

[2023] Generative AI generates next step information (e.g., "Please finely chop the tomatoes").

[2024] Step 5:

[2025] The server receives the generated procedure information and transmits it to the terminal.

[2026] Step 6:

[2027] The device will provide the user with next steps via voice or text.

[2028] Step 7:

[2029] The user follows the instructions and sends the request again if the next step is required.

[2030] Providing professional technical information

[2031] Step 1:

[2032] A user submits a request for pro technical information on a recipe (e.g., "What are some pro tips for this recipe?").

[2033] Step 2:

[2034] The device sends a request to the server.

[2035] Step 3:

[2036] The server queries a professional technical information database and provides the information to the generating artificial intelligence.

[2037] Step 4:

[2038] Generative AI optimizes technical information and generates specific advice (e.g., "Professionals recommend blanching tomatoes before chopping them to make them taste even better").

[2039] Step 5:

[2040] The server receives the optimized technical information and sends it to the terminal.

[2041] Step 6:

[2042] The terminal displays or audibly explains technical information to the user.

[2043] Healthy menu suggestions

[2044] Step 1:

[2045] If a user wants to know about healthy meals, they send a request (e.g., "Tell me about healthy meals").

[2046] Step 2:

[2047] The device sends a request to the server.

[2048] Step 3:

[2049] The server retrieves the user's health information and nutritional balance information from the database and provides it to the generating artificial intelligence.

[2050] Step 4:

[2051] Generative artificial intelligence generates nutritionally balanced menus based on health information (e.g., "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup").

[2052] Step 5:

[2053] The server receives the generated menu information and transmits it to the terminal.

[2054] Step 6:

[2055] The terminal displays the menu information to the user.

[2056] The above are the specific processing steps of the cooking assistance system that uses interactive generative artificial intelligence.

[2057] Example 1

[2058] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2059] Current cooking assistance systems often fail to adequately address the individual needs and circumstances of users. Specifically, they lack customized recipes and assistance that reflect the user's health status, food preferences, allergy information, etc. They also lack real-time assistance needed during cooking and a means to easily obtain professional technical information. Furthermore, there is a lack of healthy menu suggestions based on the user's health information, so more personalized assistance is needed.

[2060] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2061] In this invention, the server includes means for inputting user information and transmitting it to the server, means for saving the input user information, means for having the AI ​​generate a recipe based on specified ingredients and providing the recipe to the user, means for providing the user with the next steps in real time by voice or text when the user actually starts cooking and requests the next steps, means for obtaining professional technical information for a specific recipe, optimizing it, and providing it to the user, and means for having the AI ​​generate a healthy menu based on health-related information and providing the menu to the user. This makes it possible to provide customized cooking support and real-time support that meets the individual needs of the user, as well as professional technical information and healthy menu suggestions.

[2062] "User information" is a general term for data related to individual needs and circumstances, such as a user's name, age, food preferences, allergy information, and health information.

[2063] A "server" is a computer system that receives and stores information sent by a user, sends requests to the generating artificial intelligence, processes the results, and provides them to the user.

[2064] A "terminal" is a device through which a user inputs information, makes requests, and receives and displays responses from a server.

[2065] "Generative AI" is an AI model that generates optimal recipes, procedures, technical information, and menus based on a user's requests and profile.

[2066] A "recipe" is a list of instructions or ingredients for cooking a particular dish.

[2067] "Real-time assistance" refers to assistance that provides the user with the next cooking steps on the spot as they proceed with their cooking.

[2068] "Professional technical information" is information about the techniques and tips of professional chefs.

[2069] A "healthy menu" refers to a meal plan that takes into consideration the user's health and nutritional balance.

[2070] The present invention is a cooking assistance system that uses interactive generative artificial intelligence to provide users with customized cooking recipes, cooking procedures, professional technical information, and healthy menus. Specific embodiments of the present invention are described below.

[2071] The system mainly consists of a server, a terminal, and a user. The server collects user information, stores and processes data, and runs generative AI models. The terminal receives input from the user, communicates with the server, and displays information to the user.

[2072] Hardware and software used

[2073] Hardware:

[2074] Server: A high-performance computer (e.g., a cloud service server)

[2075] Devices: smartphones, tablets, computers

[2076] software:

[2077] Generative AI models: such as OpenAI's GPT-3

[2078] Database: MySQL, MongoDB

[2079] Communication method: REST API, WebSocket

[2080] Front-end technologies: HTML, JavaScript, Swift

[2081] Collection of User Information

[2082] When the user first starts the app, they enter their name, age, food preferences, allergy information, and health information on the input screen on their device. The entered information is sent from the device to the server and stored in a database on the server. This creates a foundation for providing support customized for each user.

[2083] Providing cooking recipes

[2084] If a user wants to know about recipes that use a specific ingredient, they can make a request, for example, "Tell me recipes that use tomatoes." This request is sent from the device to the server, which then uses the user's profile information and the request to generate the optimal recipe for the generative AI model. The generative AI model then sends the generated recipe information back to the server, which then sends it to the device and displays it to the user.

[2085] Example: "Tomato and basil pasta"

[2086] Real-time assistance with cooking procedures

[2087] When a user actually starts cooking and wants to know the next step, they can make a request, for example, "Tell me the next step." This request is sent from the device to the server, and the server has the generative AI model generate the next step and send the result back to the server. The server then sends the step information to the device, and the device provides the step to the user via voice or text.

[2088] Example: "Please chop the tomatoes finely."

[2089] Providing professional technical information

[2090] If a user wants to know professional technical information for a particular recipe, they can make a request, for example, "Tell me the professional tips for this recipe." This request is sent from the device to the server, which queries a database of professional technical information and optimizes that information for the generative AI model. The optimized technical information is then sent from the server to the device, which then provides that information to the user.

[2091] Example: "Professionals blanch tomatoes before chopping them to enhance their flavor."

[2092] Healthy menu suggestions

[2093] If a user wants to know about healthy menus, they can make a request, for example, "Tell me about healthy menus." This request is sent from the device to the server, and the server uses the user's health checkup data and nutritional balance information to generate an optimal menu using the generative AI model. The generated menu is then sent to the device via the server and displayed to the user.

[2094] Example: "Breakfast: Oatmeal and fruit, Lunch: Grilled salmon and salad, Dinner: Chicken and vegetable soup."

[2095] As described above, the cooking assistance system of the present invention can provide users with customized recipes and procedures, professional technical information, and healthy menus, thereby providing practical and effective support.

[2096] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2097] Step 1:

[2098] Collection of User Information

[2099] Input: The user enters their name, age, food preferences, allergy information, and health information into the terminal.

[2100] Processing: The terminal sends the entered information to the server, which receives the information and processes it to save it in a database. Specifically, the data sent from the terminal undergoes validation checks on the server side before being saved in a database (for example, MySQL or MongoDB).

[2101] Output: Save to database, and user information is stored on the server.

[2102] Specific operation: When the user presses the send button, the message "Sent. Thank you." is displayed on the terminal.

[2103] Step 2:

[2104] Providing cooking recipes

[2105] Input: The user types a request into the terminal, such as "Tell me some recipes using tomatoes."

[2106] Processing: The device sends this request to the server, which then sends the received request and the user's profile information to the generative AI model (e.g., OpenAI's GPT-3). The generative AI model generates the optimal recipe based on the prompt, which includes the user's profile information (such as name, age, and food preferences).

[2107] Output: The recipe information generated by the generative AI model is sent back to the server, which then sends it to the device and displays it to the user.

[2108] Specific operation: An image of "Tomato and Basil Pasta" and recipe instructions will be displayed on the device screen.

[2109] Step 3:

[2110] Real-time assistance with cooking procedures

[2111] Input: The user types a request into the terminal, such as "Tell me what to do next."

[2112] Processing: The device sends a request to the server, which then sends it to the generative AI model to generate the next step. The prompt contains the current step and the next step. The generative AI model generates the next step and sends the result back to the server.

[2113] Output: The generated next step information is sent from the server to the terminal and provided to the user by voice or text.

[2114] Specific operation: The device will say, "Please finely chop the tomatoes."

[2115] Step 4:

[2116] Providing professional technical information

[2117] Input: A user types a request into a device, such as "Tell me some pro tips for this recipe."

[2118] Processing: The device sends a request to the server, which queries a professional technical information database to obtain information and optimizes it for the generative AI model. The prompt contains the recipe and related technical information. The generative AI model returns the optimized technical information to the server.

[2119] Output: The optimized technical information is sent from the server to the terminal and displayed or explained to the user by voice.

[2120] Specific action: The information displayed is, "Professionals blanch tomatoes before chopping them to make them taste even better."

[2121] Step 5:

[2122] Healthy menu suggestions

[2123] Input: The user inputs a request into the terminal, such as "Tell me some healthy meals."

[2124] Processing: The device sends a request to the server, which then sends a prompt to the generative AI model to generate an optimal menu based on the user's health checkup data and nutritional balance information. The generative AI model generates a menu that takes nutritional balance into consideration and returns the results to the server.

[2125] Output: The generated menu information is sent from the server to the terminal and displayed to the user.

[2126] Specific behavior: The menu will be displayed: "Breakfast: Oatmeal and fruit, Lunch: Grilled salmon and salad, Dinner: Chicken and vegetable soup."

[2127] (Application example 1)

[2128] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2129] Currently, there are a wide variety of systems that provide cooking instructions and recipes, but these systems generally do not take into account the user's individual preferences, health status, or the specific ingredients used. Even in brick-and-mortar stores, customers are rarely offered recipe suggestions based on the ingredients they have on hand, and it is difficult to obtain professional technical advice or next cooking steps in real time. As a result, users often have to go through a lot of effort to obtain the information they want, resulting in a lack of convenience.

[2130] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2131] In this invention, the server includes a means for inputting user information and transmitting it to the server, a means for saving the input user information, and a means for causing a recipe generation AI to generate a recipe based on designated ingredients and providing the recipe to the user. This makes it possible to propose recipes based on ingredients in-store using advanced information terminals. It also makes it possible to provide the user with next steps in real time by voice or text.

[2132] Furthermore, by providing a means to acquire professional technical information, optimize it, and provide it to users, it is possible to convey professional cooking techniques and tips to users. In addition, by using health-related information to generate healthy menus using artificial intelligence, it is possible to provide these menus to users, making it possible to suggest optimal menus based on each individual's health status. This will greatly improve user convenience and provide optimal cooking support.

[2133] "Interactive generative artificial intelligence" is an artificial intelligence that collects information through dialogue with users and generates and provides appropriate information and services based on that information.

[2134] A "cooking assistance system" is a system that assists users when cooking by suggesting recipes and guiding them through cooking procedures.

[2135] "User Information" refers to personal information such as the user's name, age, food preferences, allergy information, and health information.

[2136] A "server" is a computer system used to store user information, process data, and generate recipes.

[2137] "Advanced information terminals" refer to electronic devices with advanced functions, such as smartphones and digital kiosks.

[2138] A "recipe" is a collection of information on how to make a dish, such as the ingredients to be used, their amounts, and cooking steps.

[2139] "Professional technical information" refers to information related to cooking techniques and tips possessed by professional chefs.

[2140] "Health-related information" refers to data related to maintaining and improving health, such as a user's health checkup data and nutritional balance information.

[2141] A "menu" refers to a meal combination or menu plan for a specific period of time.

[2142] The present invention relates to a cooking assistance system that uses interactive generative artificial intelligence and aims to provide optimal assistance to users when cooking. Specific embodiments will be described below.

[2143] 1. System Configuration

[2144] This system consists of a terminal for inputting user information, a server for storing and processing data, and multiple functions that utilize the generative AI model. The terminals are advanced information terminals such as smartphones and digital kiosks. The server stores user information and processes the data, and generates various data using a generative AI model (e.g., OpenAI's GPT-4).

[2145] 2. Collection of User Information

[2146] The server collects user information (such as name, age, food preferences, allergy information, and health information) sent from the device and stores it in a database, enabling it to provide customized services tailored to each individual user.

[2147] 3. Recipe Generation

[2148] When a user wants to know a recipe based on specific ingredients, the device sends the request to the server. The server uses the user information and the request to generate the optimal recipe using a generative AI model, which then sends the recipe back to the device. For example, a recipe such as "pasta with tomatoes and basil" is generated and served to the user.

[2149] 4. Providing real-time procedures

[2150] When a user wants real-time instructions for cooking a dish, the device sends a request to the server, which uses a generative AI model to generate the next steps and sends them back to the device in real time, providing specific instructions via voice or text, such as "finely chop the tomatoes."

[2151] 5. Providing professional technical information

[2152] When a user wants to know professional technical information for a particular recipe, the device sends the request to the server. The server queries a database of professional technical information, optimizes the information with a generative AI model, and provides it to the user. For example, the information provided may be, "Professionals recommend blanching the tomatoes before chopping them to enhance the flavor."

[2153] 6. Healthy meal suggestions

[2154] When a user wants to know about healthy meals, the device sends a request to the server. The server uses the user's health checkup data and nutritional balance information to generate an optimal meal plan using the generative AI model, and sends the plan back to the device. For example, a menu such as "Breakfast: Oatmeal and fruit, Lunch: Grilled salmon and salad, Dinner: Chicken and vegetable soup" may be provided.

[2155] Hardware and Software

[2156] The server is built using programming languages ​​and frameworks such as Python and Flask, and uses OpenAI's API for generative AI, with user information stored in a data storage system such as an SQLite database.

[2157] Specific examples

[2158] Examples of prompts include:

[2159] "Tell me a recipe that uses tomatoes, basil, and pasta."

[2160] "Tell me the next cooking step"

[2161] "What are some pro tips for this recipe?"

[2162] By using these prompts, users can obtain optimal recipes, cooking procedures, and specialized technical information in real time. In this way, a cooking assistance system using interactive generative AI can provide users with highly convenient and effective support.

[2163] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2164] Step 1:

[2165] Collection of User Information

[2166] Users enter their name, age, food preferences, allergy information, and health information through a device (smartphone or digital kiosk). The entered data is sent from the device to a server, which receives the data and stores it in an SQLite database. This creates the foundation for providing services customized for each user.

[2167] Step 2:

[2168] Recipe Generation

[2169] When a user wants to know a recipe based on a specific ingredient, they can input, for example, "Tell me a recipe using tomatoes" into their device. The device then sends this request to the server. The server references the user information and the request and generates the optimal recipe based on a generative AI model (for example, GPT-4). The generated recipe is then sent back from the server to the device and displayed to the user. Specifically, in response to the prompt "Tell me a recipe using tomatoes, basil, and pasta," "Pasta with tomatoes and basil" is suggested.

[2170] Step 3:

[2171] Providing real-time procedures

[2172] When a user wants real-time support for cooking steps, they can type, for example, "Tell me the next steps" into their device. Upon receiving this request, the device sends a request to the server. The server generates the next cooking steps based on a generative AI model. These steps are sent back from the server to the device and provided to the user via text or voice. Specifically, instructions such as "Please finely chop the tomatoes" are provided in real time.

[2173] Step 4:

[2174] Providing professional technical information

[2175] If a user wants to know professional technical information for a particular recipe, they can type, for example, "Tell me the pro tips for this recipe" into their device. The device then sends this request to the server. The server queries a database of professional technical information and optimizes the information using a generative AI model. The optimized information is then sent back from the server to the device and presented to the user via display or audio. For example, they might provide information such as, "Professionals blanch tomatoes before chopping them to enhance their flavor."

[2176] Step 5:

[2177] Healthy menu suggestions

[2178] If a user wants to know what a healthy menu is, they can type, for example, "Tell me what a healthy menu is." The device sends this request to the server. The server uses the user's health checkup data and nutritional balance information to have the generative AI model generate the optimal menu. This menu information is sent back from the server to the device and provided to the user. Specifically, the suggested menu might be "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup."

[2179] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2180] The present invention is a cooking assistance system that utilizes an emotion engine in addition to interactive generative artificial intelligence, and provides more personalized assistance by recognizing the user's emotions and customizing recipes and procedures based on those emotions, and adjusting the tone and content of the dialogue. Specific embodiments of the present invention are described below.

[2181] System Overview

[2182] Users can interact with the system through an interface and receive various cooking assistance. The system utilizes information from the server, the device, and the user, and also uses an emotion engine to suggest recipes, provide real-time assistance with procedures, provide professional technical information, and suggest healthy meals.

[2183] Program processing overview

[2184] The system works through user information collection, recipe generation, real-time assistance, professional technical information provision, healthy meal suggestions, as well as emotion recognition and optimization using an emotion engine.

[2185] Collection of User Information

[2186] 1. The user launches the app for the first time and is prompted to enter user information.

[2187] 2. The device displays fields for the user to enter their name, age, food preferences, allergy information, and health information.

[2188] 3. The user enters this information and presses the submit button.

[2189] 4. The terminal sends the entered information to the server.

[2190] 5. The server stores the received user information in a database.

[2191] Providing cooking recipes

[2192] 1. A user voices or texts a request for a recipe using a specific ingredient (e.g., "Tell me some recipes using tomatoes").

[2193] 2. The terminal sends the input request to the server.

[2194] 3. The server uses the AI ​​to generate the optimal recipe based on the user's profile information and request.

[2195] 4. The generation AI returns the generated recipe information to the server.

[2196] 5. The server sends the recipe information to the terminal, and the terminal displays the generated recipe (e.g., "Pasta with Tomato and Basil") for the user.

[2197] Real-time assistance with cooking procedures

[2198] 1. If a user is cooking and wants to know the next step, they can send a request via voice or text (e.g., "Tell me the next step").

[2199] 2. The device sends a request to the server.

[2200] 3. The server has the generation AI generate the next steps and send those steps back to the server.

[2201] 4. The generative AI generates next step information (e.g., "Please finely chop the tomatoes").

[2202] 5. The server receives the generated procedure information and sends it to the terminal.

[2203] 6. The device will provide the user with next steps via voice or text.

[2204] Providing professional technical information

[2205] 1. A user submits a request for pro technical information on a recipe (e.g., "What are some pro tips for this recipe?").

[2206] 2. The device sends a request to the server.

[2207] 3. The server queries a professional technical information database and provides that information to the generation AI.

[2208] 4. Generative AI optimizes technical information and generates specific advice (e.g., "Professionals recommend blanching tomatoes before chopping them to make them taste even better").

[2209] 5. The server receives the optimized technical information and sends it to the device.

[2210] 6. The device displays or audibly explains technical information to the user.

[2211] Healthy menu suggestions

[2212] 1. If a user wants to know about healthy meals, they send a request (e.g., "Tell me about healthy meals").

[2213] 2. The device sends a request to the server.

[2214] 3. The server retrieves the user's health checkup data and nutritional balance information from the database and provides it to the generation AI.

[2215] 4. Generative AI generates nutritionally balanced menus based on health information (e.g., "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup").

[2216] 5. The server receives the generated menu information and sends it to the terminal.

[2217] 6. The device displays the menu information to the user.

[2218] Utilizing the Emotion Engine

[2219] 1. If the user's emotions change during a conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion.

[2220] 2. The device sends the recognized emotion data to the server.

[2221] 3. The server provides the generative AI with emotional data and issues instructions to customize the recipe, steps, and dialogue content.

[2222] For example, if a user is feeling stressed about cooking, the emotion engine will recognize this and the generative AI will suggest recipes that are easy and quick to make. This allows users to receive optimal cooking assistance tailored to their mood and situation.

[2223] In this way, a cooking assistance system that combines interactive generative artificial intelligence and an emotion engine can provide users with customized recipes and procedures, professional techniques, and healthy menus, and can also respond to their emotions, providing more personalized assistance.

[2224] The processing flow will be explained below.

[2225] Processing flow of a cooking support system using an emotion engine

[2226] Collection of User Information

[2227] Step 1:

[2228] The user launches the app for the first time and is presented with a screen to enter user information.

[2229] Step 2:

[2230] The device will present the user with fields to enter their name, age, food preferences, allergy information, and health information.

[2231] Step 3:

[2232] The user enters this information and presses the send button.

[2233] Step 4:

[2234] The terminal transmits the input information to the server.

[2235] Step 5:

[2236] The server stores the received user information in a database.

[2237] Providing cooking recipes

[2238] Step 1:

[2239] The user voices or types in a request for a recipe using a specific ingredient (e.g., "Tell me some recipes using tomatoes").

[2240] Step 2:

[2241] The terminal transmits the input request to the server.

[2242] Step 3:

[2243] The server references the user's profile information from the database and sends the request to the generation artificial intelligence.

[2244] Step 4:

[2245] Generative AI generates the optimal recipe based on the user's profile and specified ingredients.

[2246] Step 5:

[2247] The server receives the generated recipe information and transmits it to the terminal.

[2248] Step 6:

[2249] The device displays the generated recipe (e.g., "Pasta with Tomato and Basil") to the user.

[2250] Real-time assistance with cooking procedures

[2251] Step 1:

[2252] If a user is cooking and wants to know the next step, they can send a request by voice or text (e.g., "Tell me the next step").

[2253] Step 2:

[2254] The device sends a request to the server.

[2255] Step 3:

[2256] The server issues instructions to the generation artificial intelligence to generate the next step.

[2257] Step 4:

[2258] Generative AI generates next step information (e.g., "Please finely chop the tomatoes").

[2259] Step 5:

[2260] The server receives the generated procedure information and transmits it to the terminal.

[2261] Step 6:

[2262] The device will provide the user with next steps via voice or text.

[2263] Providing professional technical information

[2264] Step 1:

[2265] A user submits a request for pro technical information on a recipe (e.g., "What are some pro tips for this recipe?").

[2266] Step 2:

[2267] The device sends a request to the server.

[2268] Step 3:

[2269] The server queries a professional technical information database and provides the information to the generating artificial intelligence.

[2270] Step 4:

[2271] Generative AI optimizes technical information and generates specific advice (e.g., "Professionals recommend blanching tomatoes before chopping them to make them taste even better").

[2272] Step 5:

[2273] The server receives the optimized technical information and sends it to the terminal.

[2274] Step 6:

[2275] The terminal displays or audibly explains technical information to the user.

[2276] Healthy menu suggestions

[2277] Step 1:

[2278] If a user wants to know about healthy meals, they send a request (e.g., "Tell me about healthy meals").

[2279] Step 2:

[2280] The device sends a request to the server.

[2281] Step 3:

[2282] The server retrieves the user's health checkup data and nutritional balance information from the database and provides it to the generating artificial intelligence.

[2283] Step 4:

[2284] Generative artificial intelligence generates nutritionally balanced menus based on health information (e.g., "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup").

[2285] Step 5:

[2286] The server receives the generated menu information and transmits it to the terminal.

[2287] Step 6:

[2288] The terminal displays the menu information to the user.

[2289] Utilizing the Emotion Engine

[2290] Step 1:

[2291] If the user's emotions change during the conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion.

[2292] Step 2:

[2293] The device transmits the recognized emotion data to the server.

[2294] Step 3:

[2295] The server provides emotional data to the generative AI, which then issues instructions to customize recipes, steps, and dialogue content.

[2296] As a concrete example, if a user is feeling stressed about cooking, the emotion engine will recognize this and the generative AI will suggest recipes that are easy and quick to make. In this example, it is possible to suggest "time-saving recipes that can be made in the microwave." This allows users to receive optimal cooking support tailored to their mood and situation.

[2297] Example 2

[2298] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2299] In today's modern lifestyles, it is extremely important to provide healthy and efficient cooking assistance that accommodates the different lifestyles and food preferences of each individual user. However, conventional cooking assistance systems lack the ability to customize based on the user's emotions and health status, making it difficult to provide more personalized assistance. There is also a need to effectively incorporate technical information from experts to improve the quality of cooking. Furthermore, in situations where real-time response is required, providing prompt and appropriate information has been insufficient.

[2300] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2301] In this invention, the server includes means for inputting user information and transmitting it to the server, means for saving the input user information, means for having a generation AI generate a recipe based on specified ingredients and providing the recipe to the user, means for providing the user with real-time audio or text instructions for the next steps, means for acquiring technical information from experts and optimizing it to provide to the user, means for having the generation AI generate a healthy menu based on health-related information and providing the menu to the user, and means for analyzing the user's emotions using an emotion recognition engine and customizing the recipe, steps, and dialogue content based on the emotions. This enables personalized cooking support according to the individual needs and emotions of the user, provision of menus that accommodate health management, and high-quality cooking support utilizing the skills of experts.

[2302] "User information" refers to the name, age, food preferences, allergy information, health information, etc. entered by the user.

[2303] "Server" refers to a computer system that stores input user information, processes various data, and provides it to the generation AI.

[2304] "Specified ingredients" refers to ingredients that the user specifies to be used in creating the recipe.

[2305] "Generative AI" refers to an AI model that generates recipes, steps, menus, etc. based on user requests.

[2306] "Real-time" refers to nearly immediate response to user requests.

[2307] "Expert technical information" refers to information about professional knowledge and techniques related to cooking.

[2308] "Health-related information" refers to all data related to health and nutrition, such as a user's health checkup data and nutritional balance information.

[2309] An "emotion recognition engine" refers to a system that recognizes emotions by analyzing the user's voice and facial expressions.

[2310] "Personalized cooking assistance" refers to providing cooking assistance customized based on the user's individual information and emotions.

[2311] "Customizing dialogue content" refers to adjusting the tone and content of dialogue provided according to the user's emotions and situation.

[2312] The present invention is a cooking assistance system that utilizes interactive generative artificial intelligence (generative AI model) and an emotion engine to recognize the user's emotions and customize recipes and procedures according to those emotions, thereby providing more personalized assistance. A specific embodiment of this system is shown below.

[2313] Overall system overview

[2314] Users can interact with the system through devices such as smartphones and tablets and receive various cooking assistance. The system utilizes information from the server, devices, and users, and uses an emotion engine to suggest recipes, provide real-time assistance with procedures, provide expert technical information, and suggest healthy meals. The system mainly uses the following hardware and software:

[2315] Hardware: smartphones, tablets, servers, database servers

[2316] Software: Interactive generative AI models, emotion recognition engines, user interface applications, database management systems

[2317] Program processing overview

[2318] The system works through user information collection, recipe generation, real-time assistance, expert technical input, healthy meal suggestions, as well as emotion recognition and optimization using an emotion engine.

[2319] Collection of User Information

[2320] When a user launches the app for the first time, a screen for entering user information is displayed. The device displays fields for entering the user's name, age, food preferences, allergy information, and health information. When the user enters this information and presses the send button, the device sends the entered information to the server. The server stores the received user information in a database.

[2321] Providing cooking recipes

[2322] A user inputs a request for a recipe using a specific ingredient via voice or text. For example, if the request is "Tell me a recipe using tomatoes," the device analyzes the input request and sends it to the server. The server uses the user's profile information and the request content to have the generative AI model generate the optimal recipe. The generative AI generates recipe information, such as "pasta with tomatoes and basil," and sends it back to the server. The server then sends the generated recipe information to the device and displays it to the user.

[2323] Real-time assistance with cooking procedures

[2324] When a user wants to know the next step while cooking, they send a request by voice or text. For example, if a request is made to "tell me the next step," the device sends the request to the server. The server then has the generative AI model generate the next step and sends that step back to the server. The generative AI generates step information such as "finely chop the tomatoes," and the server sends that information to the device. The device then provides the next step to the user by voice or text.

[2325] Providing technical information from experts

[2326] A user sends a request for expert technical information about a recipe. For example, if the user requests, "Tell me the pro tips for this recipe," the device sends the request to the server. The server queries a database of expert technical information and provides that information to the generative AI model. The generative AI generates specific technical information, such as "Professionals recommend blanching tomatoes before chopping them to enhance their flavor," and sends it back to the server. The server then sends that information to the device and explains it to the user by displaying it or by voice.

[2327] Healthy menu suggestions

[2328] When a user wants to know about healthy menu options, they send a request. For example, if the request is "Tell me about healthy menu options," the device sends the request to the server. The server retrieves the user's health checkup data and nutritional balance information from a database and provides this to the generative AI model. The generative AI generates a healthy menu option such as "Breakfast: oatmeal and fruit, lunch: grilled salmon and salad, dinner: chicken and vegetable soup." The server sends the generated menu information to the device and displays it to the user.

[2329] Utilizing the Emotion Engine

[2330] If the user's emotions change during a conversation, the emotion engine analyzes the user's voice and facial expressions to recognize the emotion. The device then sends the recognized emotion data to the server, which then provides the emotion data to the generative AI model and issues instructions to customize the recipe, steps, and conversation content. For example, if the user is feeling stressed about cooking, the emotion engine will recognize this, and the generative AI model will suggest a recipe that is easy and quick to make. This allows the user to receive optimal cooking assistance tailored to their mood and situation.

[2331] Examples of prompt statements

[2332] Recipe generation prompt: "Generate a quick recipe using tomatoes. The user is stressed."

[2333] Technical information prompt: "What are some pro tips for this recipe?"

[2334] In this way, the cooking assistance system of the present invention combines interactive generative artificial intelligence with an emotion engine to provide users with customized recipes, procedures, expert techniques, and healthy menus, and can also respond to emotions, making it possible to provide more personalized assistance.

[2335] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2336] Collection of User Information

[2337] Step 1:

[2338] The user launches the app for the first time and is presented with a screen to enter user information.

[2339] What it does: The device displays fields for entering name, age, food preferences, allergy information, and health information.

[2340] Step 2:

[2341] The user enters this information and presses the send button.

[2342] Specific operation: The device receives the entered information, detects when the send button is pressed, and sends it to the server.

[2343] Input: User's personal information (name, age, food preferences, allergy information, health information)

[2344] Output: User information sent to the server

[2345] Step 3:

[2346] The terminal transmits the input information to the server.

[2347] Specific operation: The terminal sends the received user information to the server via a secure communication channel.

[2348] Input: User information (secure format)

[2349] Output: User information sent to the server

[2350] Step 4:

[2351] The server stores the received user information in a database.

[2352] Specific operation: The server stores the received user information in a database and makes it available for subsequent processes.

[2353] Input: User information sent to the server

[2354] Output: User information stored in the database

[2355] Providing cooking recipes

[2356] Step 1:

[2357] A user voices or texts a request for a recipe using a specific ingredient.

[2358] Specific action: For example, request "Tell me some recipes that use tomatoes."

[2359] Input: Request for a specific material

[2360] Output: The text or audio data of the request

[2361] Step 2:

[2362] The terminal transmits the input request to the server.

[2363] Specific operation: The device performs voice recognition and text analysis and sends the request content to the server.

[2364] Input: The text or audio data of the request

[2365] Output: The request sent to the server

[2366] Step 3:

[2367] The server generates a prompt for the generative AI model based on the user's profile information and request content.

[2368] Specific operation: The server converts the user information and request content into a prompt text.

[2369] Input: User profile information and request details

[2370] Output: Prompt sentence for generative AI model

[2371] Step 4:

[2372] The generative AI model generates recipe information based on the prompt sentence and sends it back to the server.

[2373] Specific behavior: The generative AI generates specific recipes such as "pasta with tomato and basil."

[2374] Input: prompt statement

[2375] Output: Generated recipe information

[2376] Step 5:

[2377] The server transmits the generated recipe information to the terminal.

[2378] Specific operation: The server transfers the recipe information to the device.

[2379] Input: Generated recipe information

[2380] Output: Recipe information sent to the device

[2381] Step 6:

[2382] The terminal displays the generated recipe to the user.

[2383] Specific operation: The device displays the received recipe information on the user interface.

[2384] Input: Recipe information sent to the device

[2385] Output: Recipe information displayed to the user

[2386] Real-time assistance with cooking procedures

[2387] Step 1:

[2388] If a user is cooking and wants to know the next step, they can send a reque...

Claims

1. A cooking assistance system using interactive generative artificial intelligence, A means for inputting user information and transmitting it to a server; a means for storing the entered user information; A means for generating a recipe based on designated ingredients using artificial intelligence and providing the recipe to the user; A means of providing the user with real-time next steps via voice or text; and A means to obtain, optimize, and provide professional technical information to users; A means for generating a healthy menu using artificial intelligence based on information related to health and providing the menu to a user; A system including:

2. 2. The system according to claim 1, further comprising means for a user to request the next cooking procedure by voice, and for the server to cause the generating artificial intelligence to generate the next procedure and provide it to the user by voice output.

3. 2. The system according to claim 1, further comprising means for causing the artificial intelligence to generate a healthy menu based on the user's health checkup data and nutritional balance information, and providing the menu to the user.

Images