System

Abstract

A system is provided.SOLUTION: A system comprising: means for storing recipe data in a database; means for performing user authentication; means for performing recipe search; means for displaying a recipe procedure step by step; means for suggesting alternative ingredients; means for a user to upload and share pictures of dishes; and means for displaying feedback and comments from other users.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Japanese cuisine is popular worldwide, but cooking methods and ingredients are difficult to obtain, making it difficult for many people to try making Japanese cuisine themselves. In particular, Japanese recipes often involve detailed steps, making them challenging for beginners. Furthermore, ingredients are often difficult to obtain locally, leaving alternatives in demand. At the same time, beginner cooks often lack confidence and need accurate feedback and community support. This invention aims to solve these challenges. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for storing recipe data in a database, a means for user authentication, a means for searching for recipes, a means for displaying step-by-step recipe procedures, a means for suggesting alternative ingredients, a means for users to upload and share photos of dishes, and a means for displaying feedback and comments from other users. This allows users to easily cook Japanese dishes at home, and helps them gain confidence and promote self-improvement through the community. The system also includes a means for managing session information and updating the community feed, improving the user experience.

[0006] "Recipe data" is data that includes detailed information such as cooking instructions, ingredient lists, proportions, etc.

[0007] A "database" is a system for systematically storing, managing, and searching recipe data, user information, etc.

[0008] "User authentication" is the process of verifying that a user is a legitimate user based on the user's ID and password.

[0009] "Recipe Search" is a feature that allows users to search the database using keywords and categories to find a specific recipe.

[0010] A "recipe step" is a series of tasks or operational steps required to complete a dish.

[0011] "Step by step" is a method of explaining each step in detail.

[0012] "Substitute ingredients" are other ingredients that can be used in place of a specific ingredient listed in a recipe if that ingredient is not available.

[0013] "Uploading" is the act of a user sending data from their device to a server.

[0014] "Feedback" refers to reactions such as comments and ratings provided by other users.

[0015] The "Community Feed" is a shared space between users where uploaded photos and comments are displayed in a timeline format.

[0016] "Session information" is temporary information used to manage the state of a user while the user is logged in. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The system of the present invention is mainly composed of a server, user terminals, and a user community. Each element will be explained separately below.

[0039] User Registration and Login

[0040] In order for a user to use the system, they must first register. At this time, the user enters their "user name," "email address," and "password" from their device and sends a registration request to the server. The server receives this and stores the user information in a database. Once registration is complete, the user enters their "email address" and "password" on the login screen and is authenticated. The server compares the information with that in the database and, if successful, returns session information to the user.

[0041] Search and browse recipes

[0042] When a user searches for a Japanese recipe, they enter a keyword (e.g., "sushi") into the device's search bar and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and check its details.

[0043] Step-by-step guide

[0044] For the recipe selected by the user, each cooking step can be checked on the details screen. Upon receiving a request from the device, the server retrieves the corresponding step data (including photos and videos) from the database and sends them to the device in sequence. The user proceeds with the cooking process by checking each step displayed on the screen.

[0045] Proposal of alternative materials

[0046] If a user has difficulty obtaining a particular ingredient, they need to be offered an alternative. In this case, the device sends a request for an alternative ingredient to the server. The server retrieves the relevant alternative from the database and returns it to the device. The user can then continue cooking using the alternative provided.

[0047] Food photo sharing and community support

[0048] After the dish is complete, the user takes a photo and uploads it to the server along with their comments. The server stores the photo and comments in a database and updates the community feed. This update is then reflected on other users' devices, allowing them to receive feedback and comments from other users. This promotes interaction and support among users.

[0049] Specific examples

[0050] For example, consider a case where a user wants to make "sushi." After logging in, the user enters "sushi" in the search bar and presses the search button. The server extracts recipes related to "sushi" from the database and returns the information to the device. The user selects "sushi" from the displayed list and checks the detailed information.

[0051] Next, you cook using a step-by-step guide. If you don't have some ingredients, you can request a substitute from the server and use the suggested alternative. Once your dish is complete, you can upload a photo and comments from your device to the server and share them with the community. You can receive feedback and comments from other users to improve your next cooking experience.

[0052] In this way, the system of the present invention supports users in the process of learning and enjoying Japanese cuisine, and promotes mutual interaction through the community.

[0053] The processing flow will be explained below.

[0054] User Registration and Login

[0055] User Registration

[0056] Step 1:

[0057] User: Enter your "Username", "Email address", and "Password" in the new registration form and press the registration button.

[0058] Step 2:

[0059] Device: Converts registration information into JSON format and sends an API request to the server.

[0060] Step 3:

[0061] Server: Receives the API request and checks the format and content of the input data.

[0062] Step 4:

[0063] Server: Save the new user information to the database.

[0064] Step 5:

[0065] Server: If the save is successful, it returns a success message to the terminal.

[0066] Step 6:

[0067] Terminal: Receives the success message and displays it to the user.

[0068] Log in

[0069] Step 1:

[0070] User: Enter your email address and password in the login form and click the login button.

[0071] Step 2:

[0072] Terminal: Converts login information into JSON format and sends an API request to the server.

[0073] Step 3:

[0074] Server: Receives the API request and retrieves the corresponding user information from the database.

[0075] Step 4:

[0076] Server: Compares the acquired user information with the entered information, and performs authentication if they match.

[0077] Step 5:

[0078] Server: If authentication is successful, it generates session information and returns it to the terminal.

[0079] Step 6:

[0080] Terminal: Receives session information and logs the user in.

[0081] Search and browse recipes

[0082] Recipe Search

[0083] Step 1:

[0084] User: Enter a keyword (e.g., "sushi") in the search bar and press the search button.

[0085] Step 2:

[0086] Terminal: Converts search keywords into JSON format and sends an API request to the server.

[0087] Step 3:

[0088] Server: Receives the API request and searches the database for the corresponding recipe information.

[0089] Step 4:

[0090] Server: Converts the relevant recipe information into JSON format and returns it to the device.

[0091] Step 5:

[0092] Terminal: Displays the received recipe information to the user.

[0093] View recipe details

[0094] Step 1:

[0095] User: Select and click on the recipe you are interested in from the list of recipes displayed.

[0096] Step 2:

[0097] Terminal: Convert the selected recipe ID into JSON format and send an API request to the server.

[0098] Step 3:

[0099] Server: Receives the API request and retrieves the corresponding recipe details from the database.

[0100] Step 4:

[0101] Server: Converts detailed information into JSON format and returns it to the terminal.

[0102] Step 5:

[0103] Terminal: Displays the received details to the user.

[0104] Step-by-step guide

[0105] View Guide

[0106] Step 1:

[0107] User: On a recipe detail screen, clicks a link or button that displays a step-by-step guide.

[0108] Step 2:

[0109] Terminal: Convert the recipe ID into JSON format and send an API request for step data to the server.

[0110] Step 3:

[0111] Server: Receives the API request and retrieves the corresponding step data from the database.

[0112] Step 4:

[0113] Server: Converts step data (including photos and videos) into JSON format and returns it to the device.

[0114] Step 5:

[0115] Terminal: Displays the received step data to the user.

[0116] Proposal of alternative materials

[0117] Request an alternative material

[0118] Step 1:

[0119] User: Clicks a button to request an alternative material suggestion for a particular material.

[0120] Step 2:

[0121] Terminal: Converts material information into JSON format and sends an API request to the server to request a substitute material.

[0122] Step 3:

[0123] Server: Receives the request and retrieves the relevant information on alternative materials from the database.

[0124] Step 4:

[0125] Server: Converts the alternative material information into JSON format and returns it to the terminal.

[0126] Step 5:

[0127] Terminal: Displays the received alternative material information to the user.

[0128] Food photo sharing and community support

[0129] Upload a photo

[0130] Step 1:

[0131] User: Take a photo of the dish and enter it along with a comment on the upload screen.

[0132] Step 2:

[0133] Device: Convert the photo and comment into JSON format and send an API request to the server to upload.

[0134] Step 3:

[0135] Server: Receives the request and stores the photos and comments in a database.

[0136] Community Feed Updates

[0137] Step 1:

[0138] Server: Update the community feed every time a new post is made.

[0139] Step 2:

[0140] Device: Receives updated feed information and displays it on the user's screen.

[0141] Step 3:

[0142] Users: View and interact with other users' feedback and comments.

[0143] The above are the specific processing steps for carrying out the present invention.

[0144] Example 1

[0145] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0146] With conventional recipe search systems, it was difficult for users to find the recipe they were looking for, and they lacked a function to suggest alternative ingredients when ingredients were unavailable, which often hindered the progress of cooking.In addition, interaction between users was limited, and they were unable to receive feedback or comments from other users, which made it difficult to improve their cooking skills or discover new recipes.

[0147] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0148] In this invention, the server includes means for storing user information in a database when a user registers, means for authenticating the user, means for searching and extracting recipe data from the database, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for the user to upload and share photos of the dish, means for displaying feedback and comments from other users, and means for displaying alternative ingredients on the user's terminal if the user cannot obtain a particular ingredient. This allows users to easily search and view desired recipes and receive suggestions for alternative ingredients even if the ingredients are unavailable. Furthermore, sharing photos of dishes promotes interaction within the community, and by receiving feedback and comments from other users, users can improve their cooking skills.

[0149] "User registration" refers to the process by which a user provides their own information to the system and is authenticated as a user.

[0150] "Storing in a database" refers to the persistent storage of digital information so that it can be accessed at a later time.

[0151] "User authentication" is the process by which a system verifies a user's identity and grants appropriate access privileges.

[0152] "Searching and extracting recipe data from a database" means searching for recipe information in a database based on specific conditions and extracting the corresponding data.

[0153] "Step-by-step" means showing the user steps or operations in a sequential, step-by-step manner.

[0154] "Suggesting alternative materials" means suggesting alternative materials that provide the same effect if the user cannot obtain a particular material.

[0155] "Users upload and share photos of their dishes" means that users send photos of their own dishes to the system and share the information with other users.

[0156] "Displaying feedback and comments from other users" means that the system collects opinions and reactions from other users and presents them in a visible form to the user.

[0157] "Displaying alternative materials on the user's device if the user cannot obtain a specific material" means that if the user cannot obtain the necessary material, the system will suggest alternative materials and display that information on the user's screen.

[0158] The system of the present invention is mainly composed of a user terminal, a server, and a database. How each function is implemented will be specifically explained below.

[0159] User Registration and Login

[0160] The user uses the terminal to enter their "user name," "email address," and "password," and sends this information to the server as a registration request. After the server receives the request, it connects to the database and saves the user information. Once the saving is complete, the server returns a response to the terminal indicating successful registration. The user re-enters their "email address" and "password" on the login screen and sends this to the server. The server compares the user information with the information in the database, and if authentication is successful, generates session information and returns it to the terminal.

[0161] Search and browse recipes

[0162] The user enters a keyword (e.g., "sushi") into the search bar of the device and sends a search request. The server receives the request and retrieves the corresponding recipe information from the database. The extracted recipe information is sent to the device, and the user can view a list of recipes as search results.

[0163] Step-by-step guide

[0164] When a user selects a specific recipe, the device sends the selection information to the server. The server then retrieves the step data for that recipe from the database and sends it to the device in sequence. The user can proceed with cooking by checking the displayed steps one by one.

[0165] Proposal of alternative materials

[0166] If a user cannot obtain a particular ingredient, the device sends a request for a substitute ingredient to the server. The server searches the database for the corresponding substitute and sends the obtained alternative to the device. The user can then confirm the suggested substitute and continue cooking.

[0167] Food photo sharing and community support

[0168] After the dish is completed, the user takes a photo of the dish using their device and sends an upload request along with their comments to the server. The server receives the request and stores the photo and comments in a database. The server then updates the community feed and notifies other users' devices, allowing other users to view the photos and comments and post feedback and comments.

[0169] For example, if a user wants to make "sushi," they first log in (see the steps for user registration and login), enter "sushi" in the search bar, and press the search button. The server extracts recipe information related to "sushi" from the database and returns it to the device. The user selects "sushi" from the displayed list of recipes and checks the details. Next, the user proceeds with the cooking while following the step-by-step guide, and if some ingredients are unavailable, they send a request for substitute ingredients to the server and use the suggested substitute ingredients. Once the dish is complete, they upload photos and comments from their device to the server and share them with the community. They can receive feedback and comments from other users to help them with their next cooking.

[0170] (Example of input to a generative AI model)

[0171] "For a Japanese recipe website, please write a paragraph explaining how to suggest alternative ingredients when ingredients cannot be easily sourced."

[0172] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0173] User Registration and Login

[0174] Step 1: Submit a user registration request

[0175] Input: The user enters their username, email address, and password on the device.

[0176] Action: User clicks the "Register" button.

[0177] Output: The device sends a registration request containing the entered information to the server.

[0178] Specific operation: The device converts the input form data into JSON format and sends it to the server as an HTTP POST request.

[0179] Step 2: Save user information

[0180] Input: The server receives a registration request.

[0181] Operation: The server connects to the database and stores the user information.

[0182] Output: If the save is successful, the server sends a registration success response to the device.

[0183] What happens: The server validates the data and executes an INSERT query to the database.

[0184] Step 3: Send a login request

[0185] Input: The user enters their email address and password on the login screen.

[0186] Action: User clicks the "Login" button.

[0187] Output: The device sends a login request containing this information to the server.

[0188] Specific operation: The device converts the input form data into JSON format and sends it to the server as an HTTP POST request.

[0189] Step 4: Authenticate and return session information

[0190] Input: The server receives a login request.

[0191] Operation: The server retrieves the relevant user information from the database and matches it with the input information.

[0192] Output: If authentication is successful, the server generates session information and returns it to the terminal.

[0193] Specific operation: The server executes a SELECT query, and if authentication is successful, generates a session ID, saves it in the database, and then returns the session ID as a response.

[0194] Search and browse recipes

[0195] Step 1: Submit a search request

[0196] Input: A user types a keyword (e.g., "sushi") into the device's search bar.

[0197] Action: User clicks the Search button.

[0198] Output: The terminal sends the entered keyword to the server.

[0199] Specific operation: The device converts the search keywords into JSON format and sends it to the server as an HTTP GET request.

[0200] Step 2: Extract recipe information

[0201] Input: The server receives a search request.

[0202] Operation: The server searches and extracts recipe information that matches the keywords from the database.

[0203] Output: The extracted recipe information is formatted into a recipe list and sent to the terminal.

[0204] Specific operation: The server executes a SELECT query, formats the results in JSON format, and sends them to the terminal as an HTTP response.

[0205] Step 3: Viewing search results

[0206] Input: The device receives a response containing the recipe information.

[0207] Interaction: The device displays a list of recipes to the user.

[0208] Output: User selects recipe they are interested in and checks its details.

[0209] Specific operation: The device parses the received JSON data and displays it in a list format in the user interface.

[0210] Step-by-step guide

[0211] Step 1: Request step information

[0212] Input: The user selects a specific recipe.

[0213] Operation: The device requests information about the selected recipe from the server.

[0214] Output: The device sends a step information request to the server.

[0215] Specific operation: The device sends JSON data including the recipe ID to the server as an HTTP GET request.

[0216] Step 2: Acquire step data

[0217] Input: The server receives a request for step information.

[0218] Operation: The server retrieves the relevant step data (including photos and videos) from the database.

[0219] Output: Sends the acquired step data to the terminal.

[0220] Specific operation: The server retrieves the relevant data using a SELECT query, formats it in JSON format, and returns it to the terminal as a response.

[0221] Step 3: Step-by-step view

[0222] Input: The terminal interprets the received step data.

[0223] Operation: The device displays the cooking steps to the user.

[0224] Output: The user proceeds with cooking by checking the steps displayed on the screen one by one.

[0225] What it does: The device analyzes information step by step and displays text and media content incrementally.

[0226] Proposal of alternative materials

[0227] Step 1: Submit a Substitute Material Request

[0228] Input: User confirms that a specific ingredient is unavailable.

[0229] Operation: The user submits a material substitution request from the terminal.

[0230] Output: The terminal sends the request data for the substitute material to the server.

[0231] Specific operation: The terminal converts the request content into JSON format and sends it to the server as an HTTP GET request.

[0232] Step 2: Get alternatives

[0233] Input: The server receives a request for a substitute material.

[0234] Operation: The server searches and retrieves information on the relevant alternative materials from the database.

[0235] Output: Sends the obtained information about alternative materials to the terminal.

[0236] Specific operation: The server retrieves relevant material information using a SELECT query, formats it in JSON format, and returns it to the terminal.

[0237] Step 3: View alternatives

[0238] Input: The terminal receives a response containing information about alternative ingredients.

[0239] Interaction: The terminal will display the alternative material information to the user.

[0240] Output: User reviews the proposed alternative materials and uses them if necessary.

[0241] Specific behavior: The device parses the received JSON data and displays it in the user interface in an appropriate format.

[0242] Food photo sharing and community support

[0243] Step 1: Upload your photo and comment

[0244] Input: User takes a photo of a dish.

[0245] Operation: A user sends an upload request from their device with a photo and comment.

[0246] Output: The device sends an upload request to the server.

[0247] Specific operation: The device converts the photo and comment into JSON format and sends it to the server as an HTTP POST request.

[0248] Step 2: Save the photo and comment

[0249] Input: The server receives an upload request.

[0250] Operation: The server stores the photos and comments in a database.

[0251] Output: If the save was successful, the server updates the community feed.

[0252] Specific behavior: The server validates the data and executes an INSERT query to save it to the database.

[0253] Step 3: Update your feed and notify

[0254] Input: The server updates the community feed.

[0255] Operation: The server sends notifications to other users' devices.

[0256] Output: Other users will receive notifications of updates to the feed.

[0257] Specific operation: The server converts the update notification into JSON format and sends it to other user devices as an HTTP PUSH notification.

[0258] This concludes the explanation of the specific processing steps of the overall system program.

[0259] (Application example 1)

[0260] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0261] Currently, there are a wide variety of systems for learning Japanese cuisine recipes, but only a limited number of systems provide practical information useful in the kitchen and support the cooking process. In particular, there are no systems that offer integrated functions such as recipe search, suggestions for alternative ingredients, step-by-step cooking guides, and photo sharing of dishes, as well as the ability to search and view restaurant menus. Therefore, there is a need for a system that allows users to efficiently and easily learn and perform cooking in real-world environments such as Japanese restaurants and cooking classes.

[0262] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0263] In this invention, the server includes means for storing recipe data in a database, means for authenticating users, means for searching for recipes, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, means for searching and browsing restaurant menus, and means for checking how to make dishes step by step in Japanese restaurants or cooking classes, thereby enabling users to efficiently and easily learn and practice cooking in real environments such as Japanese restaurants or cooking classes.

[0264] "Recipe data" refers to information including cooking methods, ingredients, and cooking steps.

[0265] A "database" refers to a system for systematically storing and managing information such as recipe data and user information.

[0266] "User authentication" refers to the process of verifying the information entered (email address and password) when a user logs in to a system and confirming that the user is a legitimate user.

[0267] "Recipe search" refers to the function that extracts recipe information from the database based on entered keywords and provides it to the user.

[0268] "Step-by-step display" refers to a function that breaks down the cooking steps of the selected recipe into order, allowing the user to proceed with the cooking process by checking each step.

[0269] "Substitute material" refers to another material that can perform an equivalent function if a particular material is not available.

[0270] "Means for suggesting alternative materials" refers to a function that suggests suitable alternative materials from a database when a user has difficulty obtaining a particular material.

[0271] "Means for uploading and sharing photos of food" refers to the ability for users to upload photos of food they have cooked to the system and share them with other users.

[0272] "Means for displaying feedback and comments" refers to a function that displays ratings and opinions from other users on the screen so that users can check them.

[0273] "Means for searching and viewing restaurant menus" refers to the function of searching for menus offered at restaurants such as Japanese restaurants and displaying their contents.

[0274] "Cooking classes" refers to facilities and services that provide educational environments and programs for learning how to cook.

[0275] This invention is a system that allows users to effectively learn and practice Japanese cuisine at brick-and-mortar establishments such as Japanese restaurants and cooking schools. This system is primarily composed of a server, user terminals, and a user community. Each element is described in detail below.

[0276] 1. User Registration and Login

[0277] Program Generation

[0278] The server has an API endpoint for user registration and login. The user enters their "username," "email address," and "password" from their device and sends them to the server. The server receives this and stores the user information in a database. When the user logs in, they enter their "email address" and "password," which the server authenticates. After successful authentication, session information is generated and returned to the user.

[0279] Natural language explanations

[0280] In order for a user to use the system, they must first register. The user enters their "user name," "email address," and "password" on their terminal and sends a registration request to the server. The server receives this and saves the user information in a database. The "email address" and "password" entered by the user when logging in are compared with the information in the database for authentication, and if successful, session information is returned to the user. This completes user authentication.

[0281] 2. Search and browse recipes

[0282] Program Generation

[0283] When a user searches for a Japanese recipe, they enter a keyword (e.g., "sushi") into the device's search bar and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and check its details.

[0284] Natural language explanations

[0285] When a user searches for Japanese cuisine, they enter specific keywords into the device's search bar and send a request to the server. The server then extracts the relevant recipe information from the database and sends it back to the device. The user can then select the recipe they are interested in from the displayed list and view detailed information.

[0286] 3. Step-by-step guide

[0287] Program Generation

[0288] The system displays the cooking steps for the recipe selected by the user, step by step. Specifically, the server receives the user's request, retrieves the corresponding step data from the database, and sends it to the device in sequence. The data may include photos and videos, and is displayed in a way that makes it easy for the user to understand.

[0289] Natural language explanations

[0290] A guide is provided to help users check the cooking steps for the recipe they have selected. Upon receiving a user's request, the server retrieves the corresponding step data from the database and sends it to the device in sequence. This allows the user to proceed with the cooking process while checking each step of the cooking procedure.

[0291] 4. Proposal of alternative materials

[0292] Program Generation

[0293] If a user has difficulty obtaining a particular ingredient, the user sends a request for a substitute ingredient from the device to the server, which retrieves the corresponding alternative from the database and returns it to the device. The user can then continue cooking using the suggested alternative.

[0294] Natural language explanations

[0295] If a user feels that a particular ingredient is difficult to obtain, they can send a request to the server for alternative ingredient suggestions. The server will then retrieve the relevant alternative ingredient suggestions from the database and send them back to the device, allowing the user to continue cooking using the suggested alternative ingredients.

[0296] 5. Food photo sharing and community support

[0297] Program Generation

[0298] After the dish is complete, the user takes a photo and uploads it to the server from their device along with their comments. The server stores the photo and comments in a database and updates the community feed, which also displays feedback and comments from other users.

[0299] Natural language explanations

[0300] Users take a photo of their finished dish and upload it to the server along with their comments from their device. The server stores the photo and comments in a database and updates the community feed, displaying feedback and comments from other users and encouraging interaction between users.

[0301] Specific examples

[0302] For example, consider a user who wants to make "sushi." After logging in, the user enters "sushi" in the search bar and presses the search button. The server retrieves recipes related to "sushi" from the database and returns the information to the device. The user selects "sushi" from the displayed list and checks the detailed information. Next, the user proceeds with the cooking process while following the step-by-step guide. If some of the ingredients are unavailable, the user can request alternative ingredients from the server and use the suggested substitute ingredients. Once the dish is complete, the user can upload photos and comments from the device to the server and share them with the community. The user can receive feedback and comments from other users and use them to improve their next cooking experience.

[0303] Prompt Sentence Examples

[0304] "A user launches the app and searches for 'sushi.' The app retrieves recipes related to 'sushi' from the database and displays them to the user. When the user selects a particular recipe, the app displays step-by-step cooking instructions. Also, if a particular ingredient is unavailable, the app suggests alternative ingredients."

[0305] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0306] Step 1:

[0307] User Registration and Login

[0308] Input: Username, Email Address, and Password

[0309] Operation: The device sends input data to the server.

[0310] Data processing: The server validates the data it receives and converts it into a format that can be stored in the database.

[0311] Output: The server saves the user information in the database and returns a registration success message to the terminal.

[0312] Step 2:

[0313] User authentication

[0314] Input: User's email address and password

[0315] How it works: The device sends authentication data to the server.

[0316] Data calculation: The server compares the data it receives with the information in the database.

[0317] Output: If authentication is successful, generate session information and return it to the terminal. If authentication fails, return an error message.

[0318] Step 3:

[0319] Search and browse recipes

[0320] Input: User search keyword (e.g. "sushi")

[0321] How it works: The device sends a search request to the server.

[0322] Data processing: The server searches the database based on keywords and extracts relevant recipe information.

[0323] Output: Returns search results to the device and displays a list of recipes to the user.

[0324] Step 4:

[0325] Step-by-step guide

[0326] Input: Recipe ID selected by the user

[0327] How it works: The device sends a request to the server.

[0328] Data calculation: The server retrieves the step data corresponding to the recipe ID from the database.

[0329] Output: Sends step data (including photos and videos) to the device sequentially and displays them to the user.

[0330] Step 5:

[0331] Proposal of alternative materials

[0332] Input: User request for specific material alternatives

[0333] How it works: The device sends a request to the server.

[0334] Data calculation: The server searches and extracts the relevant alternative materials from the database.

[0335] Output: Sends the proposed alternative material information to the terminal.

[0336] Step 6:

[0337] Food photo sharing and community support

[0338] Input: User-taken food photos and comments

[0339] What happens: Your device uploads the photo and comments to the server.

[0340] Data processing: The server receives the photos and comments and stores them in a database.

[0341] Output: Update the community feed and display the new information on other users' devices.

[0342] Examples of concrete examples and prompts

[0343] For example, if a user wants to make "sushi," the process would follow these steps:

[0344] Step 1: The user enters their username, email address, and password on their device and sends them to the server. The server receives them and stores them in a database.

[0345] Step 2: The user enters their email address and password for authentication. The server compares the information with the database and returns session information if authentication is successful.

[0346] Step 3: The user types "sushi" into the search bar and sends a search request to the server, which searches its database for recipes related to "sushi" and returns them to the device.

[0347] Step 4: The user requests the selected recipe from the server and receives step-by-step cooking instructions.

[0348] Step 5: If a particular ingredient is not available, a request for an alternative ingredient is sent to the server, which then suggests an alternative ingredient.

[0349] Step 6: After the dish is completed, the user uploads the photo and comment to the server and shares it with the community. They receive feedback and comments from other users.

[0350] Example prompt sentence:

[0351] "A user launches the app and searches for 'sushi.' The app retrieves recipes related to 'sushi' from the database and displays them to the user. When the user selects a particular recipe, the app displays step-by-step cooking instructions. Also, if a particular ingredient is unavailable, the app suggests alternative ingredients."

[0352] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0353] The system of the present invention includes means for storing recipe data in a database, means for user authentication, means for recipe search, means for displaying step-by-step recipe instructions, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, and an emotion engine for recognizing user emotions.

[0354] User Registration and Login

[0355] In order for a user to use the system, they must first register. The user enters their "user name," "email address," and "password" on their device and sends a registration request to the server. The server receives this and stores the user information in a database. Once registration is complete, the user enters their "email address" and "password" on the login screen and is authenticated. The server compares the information with that in the database and, if successful, returns session information to the user.

[0356] Search and browse recipes

[0357] When a user searches for a Japanese recipe, they enter a keyword (e.g., "sushi") into the device's search bar and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and check its details.

[0358] Step-by-step guide

[0359] The user can check each cooking step on the details screen of the selected recipe. The server receives a request from the device and retrieves the corresponding step data (including photos and videos) from the database and sends them to the device in sequence. The user proceeds with the cooking process by checking each step displayed on the screen.

[0360] Proposal of alternative materials

[0361] If a user has difficulty obtaining a particular ingredient, they need to be offered an alternative. In this case, the device sends a request for an alternative ingredient to the server. The server retrieves the relevant alternative from the database and returns it to the device. The user can then continue cooking using the alternative provided.

[0362] Food photo sharing and community support

[0363] After the dish is complete, the user takes a photo and uploads it to the server along with their comments. The server stores the photo and comments in a database and updates the community feed. This update is then reflected on other users' devices, allowing them to receive feedback and comments from other users. This promotes interaction and support among users.

[0364] Introducing the Emotion Engine

[0365] The app has an emotion engine that recognizes the user's emotions, and can analyze their facial expressions and text comments to suggest suitable recipes and substitute ingredients.

[0366] For example, consider a case where a user wants to make "sushi." After logging in, the user enters "sushi" in the search bar and presses the search button. The server extracts recipes related to "sushi" from the database and returns the information to the device. The user selects "sushi" from the displayed list and checks the detailed information.

[0367] Next, you cook using a step-by-step guide. If you don't have some ingredients, you can request a substitute from the server and use the suggested alternative. Once your dish is complete, you can upload a photo and comments from your device to the server and share them with the community. You can receive feedback and comments from other users to improve your next cooking experience.

[0368] Furthermore, the emotion engine can recognize emotions from the user's facial expressions and comments, suggesting more difficult recipes if the user is having fun, and easier recipes if the user is feeling down. This type of customized feedback is provided by the emotion engine.

[0369] In this way, the system of the present invention supports users in the process of learning and enjoying Japanese cuisine and promotes mutual interaction through the community. Furthermore, the introduction of an emotion engine provides more personalized services and improves user satisfaction.

[0370] The processing flow will be explained below.

[0371] Modes for implementing the invention combining emotion engines

[0372] User Registration and Login

[0373] Step 1:

[0374] User: Enter your "Username", "Email address", and "Password" in the new registration form and press the registration button.

[0375] Step 2:

[0376] Terminal: Converts the input information into JSON format and sends an API request to the server.

[0377] Step 3:

[0378] Server: Receives the API request and validates the input information.

[0379] Step 4:

[0380] Server: After verification, save the new user information to the database.

[0381] Step 5:

[0382] Server: Returns a message to the terminal indicating that saving is complete.

[0383] Step 6:

[0384] Terminal: Display a save complete message to the user.

[0385] Step 7:

[0386] User: Enter your email address and password in the login form and click the login button.

[0387] Step 8:

[0388] Terminal: Converts login information into JSON format and sends an API request to the server.

[0389] Step 9:

[0390] Server: Receives the API request, retrieves the relevant user information from the database, and collates it.

[0391] Step 10:

[0392] Server: If authentication is successful, it creates a user session and returns a session token to the device.

[0393] Step 11:

[0394] Terminal: Receives the session token and logs the user in.

[0395] Search and browse recipes

[0396] Step 1:

[0397] User: Enter a keyword (e.g., "sushi") in the search bar and press the search button.

[0398] Step 2:

[0399] Terminal: Converts search keywords into JSON format and sends an API request to the server.

[0400] Step 3:

[0401] Server: Receives the API request and searches the database for the corresponding recipe information.

[0402] Step 4:

[0403] Server: Converts the relevant recipe information into JSON format and returns it to the device.

[0404] Step 5:

[0405] Terminal: Displays the received recipe information to the user.

[0406] View detailed recipes and step-by-step guides

[0407] Step 1:

[0408] User: Select and click on the recipe you are interested in from the list of recipes displayed.

[0409] Step 2:

[0410] Terminal: Convert the selected recipe ID into JSON format and send an API request to the server.

[0411] Step 3:

[0412] Server: Receives the API request and retrieves the corresponding recipe details from the database.

[0413] Step 4:

[0414] Server: Converts detailed information into JSON format and returns it to the terminal.

[0415] Step 5:

[0416] Terminal: Displays the received details to the user.

[0417] Step 6:

[0418] User: On the recipe details screen, follow the step-by-step guide as you cook.

[0419] Step 7:

[0420] Terminal: Requests next step data from the server based on the user's progress.

[0421] Step 8:

[0422] Server: Retrieves the next step data from the database and returns it to the terminal.

[0423] Step 9:

[0424] Terminal: Display step data to the user.

[0425] Proposal of alternative materials

[0426] Step 1:

[0427] User: Clicks a button to request an alternative material suggestion for a particular material.

[0428] Step 2:

[0429] Terminal: Converts material information into JSON format and sends an API request to the server to request a substitute material.

[0430] Step 3:

[0431] Server: Receives the request and retrieves the relevant information on alternative materials from the database.

[0432] Step 4:

[0433] Server: Converts the alternative material information into JSON format and returns it to the terminal.

[0434] Step 5:

[0435] Terminal: Displays the received alternative material information to the user.

[0436] Food photo sharing and community support

[0437] Step 1:

[0438] User: Take a photo of the dish and enter it along with a comment on the upload screen.

[0439] Step 2:

[0440] Device: Convert the photo and comment into JSON format and send an API request to the server to upload.

[0441] Step 3:

[0442] Server: Receives the request and stores the photos and comments in a database.

[0443] Community Feed Updates

[0444] Step 1:

[0445] Server: Update the community feed every time a new post is made.

[0446] Step 2:

[0447] Device: Receives updated feed information and displays it on the user's screen.

[0448] Step 3:

[0449] Users: View and interact with other users' feedback and comments.

[0450] Introducing the Emotion Engine

[0451] Step 1:

[0452] User: Enter facial expressions or text comments while using the system.

[0453] Step 2:

[0454] Terminal: Sends the user's facial expression data or text comments to the emotion engine.

[0455] Step 3:

[0456] Emotion Engine: Analyzes the received data and identifies the user's emotions.

[0457] Step 4:

[0458] Server: Based on the analysis results from the emotion engine, it suggests suitable recipes and alternative ingredients.

[0459] Step 5:

[0460] On the device: Display suggestions to the user.

[0461] Step 6:

[0462] User: Proceed with cooking using suggested recipes and alternative ingredients.

[0463] For example, if a user wants to make "sushi," they enter "sushi" in the search bar and press the search button. The server retrieves recipes related to "sushi" from the database and returns them to the device. The user selects "sushi" from the displayed list and checks the details. If a substitute ingredient is needed during the cooking process, a request is sent to the server and the suggested ingredient is used. Once the dish is complete, the user can upload a photo and comment and receive feedback from the community.

[0464] Furthermore, the emotion engine recognizes the user's emotions from their facial expressions and comments, and suggests more difficult recipes if they are having fun, and easier recipes if they are feeling down, providing this kind of customized feedback.

[0465] Example 2

[0466] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0467] Previous recipe search systems had issues such as a lack of specific instructions required for users to cook a dish and a lack of suggestions for alternative ingredients. They also lacked functionality for users to share the cooking process and results with other users, and were unable to customize recipes based on the user's feelings. This led to issues such as low user satisfaction and a low rate of continued use of the system.

[0468] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0469] In this invention, the server includes means for storing recipe data in a database, means for authenticating users, means for searching for recipes, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, means for receiving requests from the terminal and retrieving corresponding data from the database, and an emotion engine for identifying and analyzing user emotions. This allows users to efficiently obtain the information necessary for cooking Japanese cuisine and enjoy cooking while interacting with other users. Furthermore, by utilizing the emotion engine, it is possible to suggest optimal recipes and alternative ingredients based on the user's emotions, thereby improving user satisfaction.

[0470] "Recipe data" refers to data that includes information such as cooking steps, necessary ingredients, quantities, cooking methods, etc.

[0471] A "database" is a system or device for efficiently storing, managing, and searching large amounts of data.

[0472] "User authentication" is the process of verifying that a user is authorized to access a system.

[0473] "Recipe search" is the process of extracting desired recipe data from a database based on conditions specified by the user.

[0474] "Step-by-step" refers to showing the recipe cooking steps step by step.

[0475] "Alternative materials" are substitute materials that can be used when a particular material is unavailable or according to user preference.

[0476] "Uploading" is the act of a user sending data from their own device to a server.

[0477] "Sharing" refers to making data uploaded by a user available for viewing and use by other users.

[0478] "Feedback" refers to reaction information such as comments and ratings provided by other users.

[0479] A "request" refers to a user requesting specific information or service from a server.

[0480] An "emotion engine" is a system or algorithm that analyzes emotions from a user's facial expressions and text and makes appropriate suggestions.

[0481] The system of the present invention is a comprehensive system that stores recipe data in a database, authenticates users, and provides multiple functions, such as recipe search, step-by-step recipe instructions, suggests alternative ingredients, and allows users to upload and share photos of their dishes. It also includes a sentiment engine that displays feedback and comments from other users and identifies and analyzes user sentiment.

[0482] System configuration

[0483] Hardware and Software Used

[0484] This system is primarily built using servers, databases, and user devices (smartphones and PCs). The software used includes a web application framework, a database management system, image analysis software, and sentiment analysis algorithms.

[0485] Servers: Includes web servers, database servers, and AI engine servers.

[0486] Database: Use SQL database or NoSQL database.

[0487] User devices: Smartphones, PCs, tablets, etc.

[0488] Software: Web application frameworks (e.g., Django, Ruby on Rails), database management systems (e.g., MySQL, PostgreSQL), image analysis software (e.g., OpenCV), sentiment analysis algorithms (e.g., IBM Watson, Google Cloud Natural Language API)

[0489] Explanation of program processing

[0490] User Registration and Login

[0491] When a user uses the system, they must first register. The user enters their "user name," "email address," and "password" from their device and sends a registration request to the server. The server receives this and stores the user information in a database. Once registration is complete, the user enters their "email address" and "password" on the login screen and is authenticated. The server compares the information with that in the database and, if successful, returns session information to the user.

[0492] Examples:

[0493] Prompt statement:

[0494] Enter your email address:

[0495] Please enter your password:

[0496] Enter your username:

[0497] Search and browse recipes

[0498] When a user searches for a recipe, they enter a keyword (e.g., "sushi") into the search bar on their device and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and view its details.

[0499] Examples:

[0500] Prompt statement:

[0501] Enter the name of the dish you want to search for recipes for:

[0502] Example: Sushi

[0503] Step-by-step guide

[0504] The user can check each cooking step on the details screen of the selected recipe. The server receives a request from the device and retrieves the corresponding step data (including photos and videos) from the database and sends them to the device in sequence. The user proceeds with the cooking process by checking each step displayed on the screen.

[0505] Proposal of alternative materials

[0506] If a user has difficulty obtaining a particular ingredient, they need to be offered an alternative. In this case, the device sends a request for an alternative ingredient to the server. The server retrieves the relevant alternative from the database and returns it to the device. The user can then continue cooking using the alternative provided.

[0507] Food photo sharing and community support

[0508] After the dish is complete, the user takes a photo and uploads it to the server along with their comments. The server stores the photo and comments in a database and updates the community feed. This update is then reflected on other users' devices, allowing them to receive feedback and comments from other users. This promotes interaction and support among users.

[0509] Introducing the Emotion Engine

[0510] The system has an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's emotions from their facial expressions and text comments, and can suggest appropriate recipes and alternative ingredients. Specifically, if a user comments, "I'm in a good mood today!", the system will suggest more difficult recipes. Conversely, if a user comments, "I'm tired today," the system will suggest easier recipes.

[0511] Specific examples

[0512] Here is a specific use case where the user wants to make sushi:

[0513] 1. User registration and login: Enter your username, email address, and password to register in the system, then log in to use the system.

[0514] 2. Recipe search: Enter "sushi" into the search bar on the device. The server retrieves recipes related to "sushi" from the database and returns the information to the device.

[0515] 3. Step-by-step guide: Users choose a sushi recipe and follow the step-by-step cooking instructions as they go through the process.

[0516] 4. Suggestion of alternative materials: If some of the required materials are not on hand, a request for alternative materials is made to the system, and the server proposes alternatives.

[0517] 5. Share your cooking photos: Once you've finished cooking, upload photos and comments to share with other users and get feedback.

[0518] 6. Use of emotion engine: Recognize emotions from the user's facial expressions and comments and suggest appropriate recipes.

[0519] In this way, the system of the present invention comprehensively supports users in the process of learning and enjoying cooking, and also promotes interaction with other users. Furthermore, the introduction of an emotion engine makes it possible to provide personalized services according to the user's emotions, thereby improving user satisfaction.

[0520] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0521] User Registration and Login

[0522] Step 1:

[0523] Input: The user enters their "username," "email address," and "password" on the device.

[0524] Processing: The terminal sends a registration request containing this information to the server.

[0525] Output: None.

[0526] Specific operation: When the user clicks the "New Registration" button, the terminal creates and sends a request to send the entered information to the server.

[0527] Step 2:

[0528] Input: The server receives a registration request from the terminal.

[0529] Processing: The server stores the received user information in a database.

[0530] Output: Registration successful message.

[0531] Specific operation: After the server saves the user information in the database, it returns a registration success message to the terminal.

[0532] Step 3:

[0533] Input: The user enters their email address and password on the login screen.

[0534] Processing: The terminal sends a login request containing this information to the server.

[0535] Output: None.

[0536] Specific operation: When you click the "Login" button on the login screen, the device creates a login request based on the information entered and sends it to the server.

[0537] Step 4:

[0538] Input: The server receives a login request.

[0539] Processing: The server checks the user information against the database and performs authentication.

[0540] Output: If authentication is successful, session information is generated and returned to the terminal.

[0541] What happens: The server checks the user's email address and password against records in its database, and if authentication is successful, generates and returns session information.

[0542] Search and browse recipes

[0543] Step 1:

[0544] Input: The user enters a keyword (e.g., "sushi") into the device's search bar.

[0545] Processing: The terminal sends a search request to the server.

[0546] Output: None.

[0547] Specific operation: When the user clicks the "Search" button, the device creates a search request based on the entered keywords and sends it to the server.

[0548] Step 2:

[0549] Input: The server receives a search request.

[0550] Processing: The server extracts the relevant recipe information from the database.

[0551] Output: Relevant recipe information.

[0552] Specific operation: The server searches the recipe database and extracts recipe information that matches the keywords in list form.

[0553] Step 3:

[0554] Input: The server sends the extracted recipe information to the terminal.

[0555] Processing: The device displays the received recipe information to the user.

[0556] Output: A list of displayed recipes.

[0557] Specific operation: The terminal displays the recipe information received from the server on the screen.

[0558] Step 4:

[0559] Input: The user selects the recipe of interest from the displayed list of recipes.

[0560] Processing: The device sends a request to the server to display the details of the selected recipe.

[0561] Output: None.

[0562] Specific operation: When the user clicks on the selected recipe, the device requests detailed information about that recipe from the server.

[0563] Step-by-step guide

[0564] Step 1:

[0565] Input: User requests cooking instructions on the recipe details screen.

[0566] Processing: The terminal sends a cooking instructions request to the server.

[0567] Output: None.

[0568] Specific operation: When the user clicks the "Start Cooking" button, the terminal creates a cooking procedure request and sends it to the server.

[0569] Step 2:

[0570] Input: The server receives a cooking instructions request.

[0571] Processing: The server retrieves cooking recipe data from the database.

[0572] Output: Cooking instructions data.

[0573] Specific operation: The server retrieves the cooking procedure data for the corresponding recipe from the database and returns it to the terminal along with photo and video information.

[0574] Step 3:

[0575] Input: The server sends the acquired cooking procedure data to the terminal.

[0576] Processing: The terminal displays the received procedure data to the user in sequence.

[0577] Output: Displayed cooking instructions.

[0578] Specific operation: The device receives cooking instructions from the server and displays them on the screen step by step. The user proceeds with the cooking process by checking each step one by one.

[0579] Proposal of alternative materials

[0580] Step 1:

[0581] Input: If a user does not have a specific ingredient available, they can submit a request for a substitute ingredient from their device.

[0582] Processing: The terminal sends a substitute material request to the server.

[0583] Output: None.

[0584] Specific operation: When the user clicks the "Search for alternative materials" button, the terminal creates a request for alternative materials and sends it to the server.

[0585] Step 2:

[0586] Input: The server receives a substitute material request.

[0587] Processing: The server retrieves the relevant alternative material information from the database.

[0588] Output: Alternative material information.

[0589] Specific operation: The server retrieves alternatives to the requested ingredients from the database (e.g., tuna as an alternative to salmon) and returns them to the terminal.

[0590] Step 3:

[0591] Input: The server sends the acquired alternative material information to the terminal.

[0592] Processing: The terminal displays the received alternative material information to the user.

[0593] Output: Displayed alternative material information.

[0594] Specific operation: The device displays the information on alternative ingredients received from the server on the screen, and the user can use it as a reference to continue cooking.

[0595] Food photo sharing and community support

[0596] Step 1:

[0597] Input: The user takes a photo of the finished dish and enters a comment.

[0598] Processing: The device sends an upload request containing the photo and comment to the server.

[0599] Output: None.

[0600] Specific operation: When the user clicks the "Upload Photo" button, the device will send the taken photo and any comments entered to the server.

[0601] Step 2:

[0602] Input: The server receives an upload request.

[0603] Processing: The server stores the photos and comments in a database.

[0604] Output: Save complete message.

[0605] Specific operation: The server saves the received photos and comments in a database and returns a save completion message to the terminal.

[0606] Step 3:

[0607] Input: The server updates the community feed based on the stored photos and comments.

[0608] Processing: The server creates new feed data and sends it to other user devices.

[0609] Output: The updated community feed.

[0610] Specific operation: The server updates the community feed based on the user's upload and notifies other users of the data.

[0611] Introducing the Emotion Engine

[0612] Step 1:

[0613] Input: User facial expressions and text comments.

[0614] Processing: The server collects emotion data from users' facial expressions and text comments.

[0615] Output: Emotion data.

[0616] How it works: When a user types a comment or uploads a photo, the system sends a request to the emotion engine to analyze the user's facial expressions and comments.

[0617] Step 2:

[0618] Input: Emotion data.

[0619] Processing: The emotion engine analyzes the collected data and identifies the user's emotional state.

[0620] Output: Sentiment analysis results.

[0621] Specific behavior: The emotion engine analyzes the collected data and identifies emotional states, such as "happy" or "tired."

[0622] Step 3:

[0623] Input: Sentiment analysis results.

[0624] Processing: The server suggests appropriate recipes and alternative ingredients based on the sentiment analysis results.

[0625] Output: Sentiment-based suggestion information.

[0626] Specific operation: The server receives the sentiment analysis results and suggests difficult or easy recipes to the user based on them.

[0627] In this way, the system of the present invention comprehensively supports users in the process of learning and enjoying cooking, and also promotes interaction with other users. Furthermore, the introduction of an emotion engine makes it possible to provide personalized services according to the user's emotions, thereby improving user satisfaction.

[0628] (Application example 2)

[0629] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0630] Conventional online recipe systems provide basic functionality for users to search for recipes and follow the instructions to cook, but this alone is insufficient. With the recent proliferation of food delivery services, there is a demand for systems that comprehensively support the process from ordering food to receiving it and enjoying it. Furthermore, there is a need for systems that can appropriately suggest alternative ingredients when ingredients are out of stock, and make personalized suggestions based on the user's emotions. Conventional systems lack such advanced functionality, making improving the user experience a challenge.

[0631] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0632] In this invention, the server includes means for storing recipe data in a database, means for user authentication, means for searching for recipes, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, means for providing step-by-step guidance from receiving to enjoying the ordered dish, means for suggesting alternative ingredients if an ingredient is out of stock at the time of ordering, and means for recognizing user emotions and providing personalized service. This allows users to enjoy the cooking process with guidance even after ordering and receiving the dish. Furthermore, by providing appropriate alternative suggestions even if an ingredient is out of stock, and by making personalized suggestions based on the user's emotions, a significant improvement in the user experience can be expected.

[0633] "Recipe data" is a collection of information stored in a database, including cooking instructions, necessary ingredients, and steps.

[0634] "User authentication" is the process of verifying the identification information (user name, password, etc.) of a system user and verifying that the user is a legitimate user.

[0635] "Recipe Search" is a system feature that allows users to search for information about specific dishes or ingredients.

[0636] "Step-by-step display" is a system function that guides users through each step of a recipe, checking them one by one.

[0637] "Alternative material suggestions" is a system function that suggests alternative materials when a user has difficulty obtaining a particular material.

[0638] "Means to upload and share photos of food" refers to a function that allows users to take photos of the food they have made and share them with other users on the system.

[0639] "Show feedback and comments" is a system feature that displays opinions and comments from other users.

[0640] "Step-by-step guidance" is a system function that provides detailed explanations and guidance for users to enjoy the food after the ordered food arrives.

[0641] An "emotion recognition engine" is a system function that analyzes emotions from a user's facial expressions and text comments and provides personalized services based on that.

[0642] "Personalized service provision" refers to the system's ability to provide users with information and suggestions that are best suited to them based on their specific interests and emotions.

[0643] The following system configuration is used as an embodiment of the present invention.

[0644] System Overview

[0645] The system includes a database, user authentication mechanism, recipe search function, step-by-step guidance function, ingredient substitution suggestion function, community function, emotion recognition engine, and personalized service provision function, providing comprehensive support that allows users to enjoy the cooking process from ordering to receiving their food.

[0646] Hardware and software used

[0647] Hardware: Smartphones, smart glasses, head-mounted displays, servers

[0648] Software: Flask (a Python web framework), SQLAlchemy (for database operations), emotion recognition engine (facial expression analysis and text analysis)

[0649] Details of each function

[0650] Building a database

[0651] The server stores recipe data in a database, which includes the name of the dish, the ingredients needed, the cooking instructions, and photos or videos. The database is operated using SQLAlchemy.

[0652] User authentication

[0653] In order for a user to use the system, they must register and log in. The authentication information (username, email address, password) is verified by the server, and session information is generated after authentication.

[0654] Recipe Search

[0655] The user enters keywords into the search bar and sends a recipe search request to the server. The server searches the database for the relevant recipe and returns it to the device. For example, it is possible to search for the keyword "sushi."

[0656] Step-by-step guide

[0657] The recipe details screen for a user's selected recipe displays step-by-step cooking instructions, each supplemented with photos and videos to guide the user through the cooking process.

[0658] Proposal of alternative materials

[0659] If some of the ingredients are unavailable, the user can request an alternative, and the server will retrieve the appropriate alternative from the database and return it to the user's device.

[0660] Community Features

[0661] Once the dish is complete, the user can take a photo and upload it along with a comment, which the server stores in a database and displays to other users.

[0662] Emotion Recognition Engine

[0663] Emotions are analyzed from the user's facial expressions and text comments, allowing the server to adjust the difficulty of recipes and make personalized suggestions based on the user's emotional state.

[0664] Specific examples

[0665] For example, if a user wants to make "sushi," they follow these steps: They log in to the application and type "sushi" into the search bar to search for related recipes. They select a sushi recipe that appears and follow a step-by-step guide to cook it. If some of the ingredients are unavailable, they request alternative ingredients and accept suggestions from the server. Once cooking is complete, they take a photo of the dish and share it with the community along with a comment. Furthermore, an emotion recognition engine analyzes the user's facial expressions and comments to see if they are enjoying the dish, and can suggest a recipe with an adjusted difficulty for next time.

[0666] Prompt Sentence Examples

[0667] "Just search for the name of the recipe you're interested in and get detailed step-by-step instructions on how to make it. For example, 'Sushi' or 'Curry'."

[0668] In this way, the system of the present invention can assist users in the process of enjoying cooking and improve the user experience.

[0669] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0670] Step 1:

[0671] A user accesses the system and registers.

[0672] Input: The user enters their "username," "email address," and "password" from their device and sends a registration request to the server.

[0673] Processing: The server stores the received information in a database.

[0674] Output: The server returns a registration completion message to the user terminal.

[0675] Specific operation: A user opens the application and accesses the new registration screen. They enter the required information and press the "Register" button. The server receives this and saves the user information to the database using an SQL query.

[0676] Step 2:

[0677] A user logs in with an existing account.

[0678] Input: The user enters their email address and password on the device and sends a login request to the server.

[0679] Processing: The server checks the information in its database and performs authentication.

[0680] Output: If the authentication is successful, the server generates session information and returns it to the user terminal.

[0681] Specific operation: The user goes to the login screen, enters the required authentication information, and presses the "Login" button. The server receives this and compares it with the information in the database. If the comparison is successful, the server generates session information and returns it to the user's device.

[0682] Step 3:

[0683] A user searches for a recipe.

[0684] Input: The user enters a keyword (e.g., "sushi") into the search bar of the device and sends a search request to the server.

[0685] Processing: The server extracts the relevant recipe information from the database.

[0686] Output: The server returns a list of search results to the user's device.

[0687] How it works: The user enters keywords into the search bar and presses the "Search" button. The server executes an SQL query to retrieve related recipe information from the database and returns the results to the user's device.

[0688] Step 4:

[0689] The user reviews the recipe details and follows the step-by-step instructions.

[0690] Input: Request details of the recipe selected by the user.

[0691] Processing: The server retrieves data for each step (including photos and videos) from the database.

[0692] Output: The server sends step-by-step instructions to the user terminal sequentially.

[0693] Specific operation: The user selects a recipe of interest from the displayed list of recipes and presses the details button. The server retrieves the corresponding step data from the database and sequentially sends it to the user's terminal.

[0694] Step 5:

[0695] If materials are unavailable, suggest alternative materials.

[0696] Input: The user submits a request for a substitute material from the terminal.

[0697] Processing: The server retrieves data on the corresponding alternative materials from the database.

[0698] Output: The server returns the alternative material suggestions to the user terminal.

[0699] Specific operation: If a user does not have a specific ingredient, they request an alternative to that ingredient. The server executes an SQL query to retrieve the alternative ingredient information from the database and returns it to the user's device.

[0700] Step 6:

[0701] Users upload photos of their dishes and share them with the community.

[0702] Input: After the user has finished cooking, they take a photo and send an upload request to the server along with their comments.

[0703] Processing: The server stores the received photos and comments in a database and updates the community feed.

[0704] Output: The server reflects the updated feed information to the user's terminal.

[0705] What it does: Users use the application's "Photo Upload" feature to submit photos of their finished dishes along with their comments. The server receives these, stores them in a database, and updates the community feed, which also displays feedback and comments from other users.

[0706] Step 7:

[0707] Recognize user emotions using an emotion engine.

[0708] Input: The user's facial expression data and text comments are taken as input.

[0709] Processing: The emotion engine analyzes the input data and determines the user's emotional state.

[0710] Output: Display recipe difficulty adjustment and personalized serving suggestions based on emotions.

[0711] Specific operation: The user's facial expression is captured by a camera or a comment is entered. The emotion engine analyzes this data to determine their emotional state (happy, sad, excited, etc.) and suggests the most appropriate service or recipe based on the results. For example, if the user looks happy, it will suggest a more difficult recipe, and conversely, if the user looks depressed, it will suggest an easy recipe.

[0712] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0713] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0714] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0715] [Second embodiment]

[0716] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0717] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0718] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0719] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0720] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0721] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0722] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0723] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0724] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0725] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0726] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0727] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0728] The system of the present invention is mainly composed of a server, user terminals, and a user community. Each element will be explained separately below.

[0729] User Registration and Login

[0730] In order for a user to use the system, they must first register. At this time, the user enters their "user name," "email address," and "password" from their device and sends a registration request to the server. The server receives this and stores the user information in a database. Once registration is complete, the user enters their "email address" and "password" on the login screen and is authenticated. The server compares the information with that in the database and, if successful, returns session information to the user.

[0731] Search and browse recipes

[0732] When a user searches for a Japanese recipe, they enter a keyword (e.g., "sushi") into the device's search bar and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and check its details.

[0733] Step-by-step guide

[0734] For the recipe selected by the user, each cooking step can be checked on the details screen. Upon receiving a request from the device, the server retrieves the corresponding step data (including photos and videos) from the database and sends them to the device in sequence. The user proceeds with the cooking process by checking each step displayed on the screen.

[0735] Proposal of alternative materials

[0736] If a user has difficulty obtaining a particular ingredient, they need to be offered an alternative. In this case, the device sends a request for an alternative ingredient to the server. The server retrieves the relevant alternative from the database and returns it to the device. The user can then continue cooking using the alternative provided.

[0737] Food photo sharing and community support

[0738] After the dish is complete, the user takes a photo and uploads it to the server along with their comments. The server stores the photo and comments in a database and updates the community feed. This update is then reflected on other users' devices, allowing them to receive feedback and comments from other users. This promotes interaction and support among users.

[0739] Specific examples

[0740] For example, consider a case where a user wants to make "sushi." After logging in, the user enters "sushi" in the search bar and presses the search button. The server extracts recipes related to "sushi" from the database and returns the information to the device. The user selects "sushi" from the displayed list and checks the detailed information.

[0741] Next, you cook using a step-by-step guide. If you don't have some ingredients, you can request a substitute from the server and use the suggested alternative. Once your dish is complete, you can upload a photo and comments from your device to the server and share them with the community. You can receive feedback and comments from other users to improve your next cooking experience.

[0742] In this way, the system of the present invention supports users in the process of learning and enjoying Japanese cuisine, and promotes mutual interaction through the community.

[0743] The processing flow will be explained below.

[0744] User Registration and Login

[0745] User Registration

[0746] Step 1:

[0747] User: Enter your "Username", "Email address", and "Password" in the new registration form and press the registration button.

[0748] Step 2:

[0749] Device: Converts registration information into JSON format and sends an API request to the server.

[0750] Step 3:

[0751] Server: Receives the API request and checks the format and content of the input data.

[0752] Step 4:

[0753] Server: Save the new user information to the database.

[0754] Step 5:

[0755] Server: If the save is successful, it returns a success message to the terminal.

[0756] Step 6:

[0757] Terminal: Receives the success message and displays it to the user.

[0758] Log in

[0759] Step 1:

[0760] User: Enter your email address and password in the login form and click the login button.

[0761] Step 2:

[0762] Terminal: Converts login information into JSON format and sends an API request to the server.

[0763] Step 3:

[0764] Server: Receives the API request and retrieves the corresponding user information from the database.

[0765] Step 4:

[0766] Server: Compares the acquired user information with the entered information, and performs authentication if they match.

[0767] Step 5:

[0768] Server: If authentication is successful, it generates session information and returns it to the terminal.

[0769] Step 6:

[0770] Terminal: Receives session information and logs the user in.

[0771] Search and browse recipes

[0772] Recipe Search

[0773] Step 1:

[0774] User: Enter a keyword (e.g., "sushi") in the search bar and press the search button.

[0775] Step 2:

[0776] Terminal: Converts search keywords into JSON format and sends an API request to the server.

[0777] Step 3:

[0778] Server: Receives the API request and searches the database for the corresponding recipe information.

[0779] Step 4:

[0780] Server: Converts the relevant recipe information into JSON format and returns it to the device.

[0781] Step 5:

[0782] Terminal: Displays the received recipe information to the user.

[0783] View recipe details

[0784] Step 1:

[0785] User: Select and click on the recipe you are interested in from the list of recipes displayed.

[0786] Step 2:

[0787] Terminal: Convert the selected recipe ID into JSON format and send an API request to the server.

[0788] Step 3:

[0789] Server: Receives the API request and retrieves the corresponding recipe details from the database.

[0790] Step 4:

[0791] Server: Converts detailed information into JSON format and returns it to the terminal.

[0792] Step 5:

[0793] Terminal: Displays the received details to the user.

[0794] Step-by-step guide

[0795] View Guide

[0796] Step 1:

[0797] User: On a recipe detail screen, clicks a link or button that displays a step-by-step guide.

[0798] Step 2:

[0799] Terminal: Convert the recipe ID into JSON format and send an API request for step data to the server.

[0800] Step 3:

[0801] Server: Receives the API request and retrieves the corresponding step data from the database.

[0802] Step 4:

[0803] Server: Converts step data (including photos and videos) into JSON format and returns it to the device.

[0804] Step 5:

[0805] Terminal: Displays the received step data to the user.

[0806] Proposal of alternative materials

[0807] Request an alternative material

[0808] Step 1:

[0809] User: Clicks a button to request an alternative material suggestion for a particular material.

[0810] Step 2:

[0811] Terminal: Converts material information into JSON format and sends an API request to the server to request a substitute material.

[0812] Step 3:

[0813] Server: Receives the request and retrieves the relevant information on alternative materials from the database.

[0814] Step 4:

[0815] Server: Converts the alternative material information into JSON format and returns it to the terminal.

[0816] Step 5:

[0817] Terminal: Displays the received alternative material information to the user.

[0818] Food photo sharing and community support

[0819] Upload a photo

[0820] Step 1:

[0821] User: Take a photo of the dish and enter it along with a comment on the upload screen.

[0822] Step 2:

[0823] Device: Convert the photo and comment into JSON format and send an API request to the server to upload.

[0824] Step 3:

[0825] Server: Receives the request and stores the photos and comments in a database.

[0826] Community Feed Updates

[0827] Step 1:

[0828] Server: Update the community feed every time a new post is made.

[0829] Step 2:

[0830] Device: Receives updated feed information and displays it on the user's screen.

[0831] Step 3:

[0832] Users: View and interact with other users' feedback and comments.

[0833] The above are the specific processing steps for carrying out the present invention.

[0834] Example 1

[0835] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0836] With conventional recipe search systems, it was difficult for users to find the recipe they were looking for, and they lacked a function to suggest alternative ingredients when ingredients were unavailable, which often hindered the progress of cooking.In addition, interaction between users was limited, and they were unable to receive feedback or comments from other users, which made it difficult to improve their cooking skills or discover new recipes.

[0837] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0838] In this invention, the server includes means for storing user information in a database when a user registers, means for authenticating the user, means for searching and extracting recipe data from the database, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for the user to upload and share photos of the dish, means for displaying feedback and comments from other users, and means for displaying alternative ingredients on the user's terminal if the user cannot obtain a particular ingredient. This allows users to easily search and view desired recipes and receive suggestions for alternative ingredients even if the ingredients are unavailable. Furthermore, sharing photos of dishes promotes interaction within the community, and by receiving feedback and comments from other users, users can improve their cooking skills.

[0839] "User registration" refers to the process by which a user provides their own information to the system and is authenticated as a user.

[0840] "Storing in a database" refers to the persistent storage of digital information so that it can be accessed at a later time.

[0841] "User authentication" is the process by which a system verifies a user's identity and grants appropriate access privileges.

[0842] "Searching and extracting recipe data from a database" means searching for recipe information in a database based on specific conditions and extracting the corresponding data.

[0843] "Step-by-step" means showing the user steps or operations in a sequential, step-by-step manner.

[0844] "Suggesting alternative materials" means suggesting alternative materials that provide the same effect if the user cannot obtain a particular material.

[0845] "Users upload and share photos of their dishes" means that users send photos of their own dishes to the system and share the information with other users.

[0846] "Displaying feedback and comments from other users" means that the system collects opinions and reactions from other users and presents them in a visible form to the user.

[0847] "Displaying alternative materials on the user's device if the user cannot obtain a specific material" means that if the user cannot obtain the necessary material, the system will suggest alternative materials and display that information on the user's screen.

[0848] The system of the present invention is mainly composed of a user terminal, a server, and a database. How each function is implemented will be specifically explained below.

[0849] User Registration and Login

[0850] The user uses the terminal to enter their "user name," "email address," and "password," and sends this information to the server as a registration request. After the server receives the request, it connects to the database and saves the user information. Once the saving is complete, the server returns a response to the terminal indicating successful registration. The user re-enters their "email address" and "password" on the login screen and sends this to the server. The server compares the user information with the information in the database, and if authentication is successful, generates session information and returns it to the terminal.

[0851] Search and browse recipes

[0852] The user enters a keyword (e.g., "sushi") into the search bar of the device and sends a search request. The server receives the request and retrieves the corresponding recipe information from the database. The extracted recipe information is sent to the device, and the user can view a list of recipes as search results.

[0853] Step-by-step guide

[0854] When a user selects a specific recipe, the device sends the selection information to the server. The server then retrieves the step data for that recipe from the database and sends it to the device in sequence. The user can proceed with cooking by checking the displayed steps one by one.

[0855] Proposal of alternative materials

[0856] If a user cannot obtain a particular ingredient, the device sends a request for a substitute ingredient to the server. The server searches the database for the corresponding substitute and sends the obtained alternative to the device. The user can then confirm the suggested substitute and continue cooking.

[0857] Food photo sharing and community support

[0858] After the dish is completed, the user takes a photo of the dish using their device and sends an upload request along with their comments to the server. The server receives the request and stores the photo and comments in a database. The server then updates the community feed and notifies other users' devices, allowing other users to view the photos and comments and post feedback and comments.

[0859] For example, if a user wants to make "sushi," they first log in (see the steps for user registration and login), enter "sushi" in the search bar, and press the search button. The server extracts recipe information related to "sushi" from the database and returns it to the device. The user selects "sushi" from the displayed list of recipes and checks the details. Next, the user proceeds with the cooking while following the step-by-step guide, and if some ingredients are unavailable, they send a request for substitute ingredients to the server and use the suggested substitute ingredients. Once the dish is complete, they upload photos and comments from their device to the server and share them with the community. They can receive feedback and comments from other users to help them with their next cooking.

[0860] (Example of input to a generative AI model)

[0861] "For a Japanese recipe website, please write a paragraph explaining how to suggest alternative ingredients when ingredients cannot be easily sourced."

[0862] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0863] User Registration and Login

[0864] Step 1: Submit a user registration request

[0865] Input: The user enters their username, email address, and password on the device.

[0866] Action: User clicks the "Register" button.

[0867] Output: The device sends a registration request containing the entered information to the server.

[0868] Specific operation: The device converts the input form data into JSON format and sends it to the server as an HTTP POST request.

[0869] Step 2: Save user information

[0870] Input: The server receives a registration request.

[0871] Operation: The server connects to the database and stores the user information.

[0872] Output: If the save is successful, the server sends a registration success response to the device.

[0873] What happens: The server validates the data and executes an INSERT query to the database.

[0874] Step 3: Send a login request

[0875] Input: The user enters their email address and password on the login screen.

[0876] Action: User clicks the "Login" button.

[0877] Output: The device sends a login request containing this information to the server.

[0878] Specific operation: The device converts the input form data into JSON format and sends it to the server as an HTTP POST request.

[0879] Step 4: Authenticate and return session information

[0880] Input: The server receives a login request.

[0881] Operation: The server retrieves the relevant user information from the database and matches it with the input information.

[0882] Output: If authentication is successful, the server generates session information and returns it to the terminal.

[0883] Specific operation: The server executes a SELECT query, and if authentication is successful, generates a session ID, saves it in the database, and then returns the session ID as a response.

[0884] Search and browse recipes

[0885] Step 1: Submit a search request

[0886] Input: A user types a keyword (e.g., "sushi") into the device's search bar.

[0887] Action: User clicks the Search button.

[0888] Output: The terminal sends the entered keyword to the server.

[0889] Specific operation: The device converts the search keywords into JSON format and sends it to the server as an HTTP GET request.

[0890] Step 2: Extract recipe information

[0891] Input: The server receives a search request.

[0892] Operation: The server searches and extracts recipe information that matches the keywords from the database.

[0893] Output: The extracted recipe information is formatted into a recipe list and sent to the terminal.

[0894] Specific operation: The server executes a SELECT query, formats the results in JSON format, and sends them to the terminal as an HTTP response.

[0895] Step 3: Viewing search results

[0896] Input: The device receives a response containing the recipe information.

[0897] Interaction: The device displays a list of recipes to the user.

[0898] Output: User selects recipe they are interested in and checks its details.

[0899] Specific operation: The device parses the received JSON data and displays it in a list format in the user interface.

[0900] Step-by-step guide

[0901] Step 1: Request step information

[0902] Input: The user selects a specific recipe.

[0903] Operation: The device requests information about the selected recipe from the server.

[0904] Output: The device sends a step information request to the server.

[0905] Specific operation: The device sends JSON data including the recipe ID to the server as an HTTP GET request.

[0906] Step 2: Acquire step data

[0907] Input: The server receives a request for step information.

[0908] Operation: The server retrieves the relevant step data (including photos and videos) from the database.

[0909] Output: Sends the acquired step data to the terminal.

[0910] Specific operation: The server retrieves the relevant data using a SELECT query, formats it in JSON format, and returns it to the terminal as a response.

[0911] Step 3: Step-by-step view

[0912] Input: The terminal interprets the received step data.

[0913] Operation: The device displays the cooking steps to the user.

[0914] Output: The user proceeds with cooking by checking the steps displayed on the screen one by one.

[0915] What it does: The device analyzes information step by step and displays text and media content incrementally.

[0916] Proposal of alternative materials

[0917] Step 1: Submit a Substitute Material Request

[0918] Input: User confirms that a specific ingredient is unavailable.

[0919] Operation: The user submits a material substitution request from the terminal.

[0920] Output: The terminal sends the request data for the substitute material to the server.

[0921] Specific operation: The terminal converts the request content into JSON format and sends it to the server as an HTTP GET request.

[0922] Step 2: Get alternatives

[0923] Input: The server receives a request for a substitute material.

[0924] Operation: The server searches and retrieves information on the relevant alternative materials from the database.

[0925] Output: Sends the obtained information about alternative materials to the terminal.

[0926] Specific operation: The server retrieves relevant material information using a SELECT query, formats it in JSON format, and returns it to the terminal.

[0927] Step 3: View alternatives

[0928] Input: The terminal receives a response containing information about alternative ingredients.

[0929] Interaction: The terminal will display the alternative material information to the user.

[0930] Output: User reviews the proposed alternative materials and uses them if necessary.

[0931] Specific behavior: The device parses the received JSON data and displays it in the user interface in an appropriate format.

[0932] Food photo sharing and community support

[0933] Step 1: Upload your photo and comment

[0934] Input: User takes a photo of a dish.

[0935] Operation: A user sends an upload request from their device with a photo and comment.

[0936] Output: The device sends an upload request to the server.

[0937] Specific operation: The device converts the photo and comment into JSON format and sends it to the server as an HTTP POST request.

[0938] Step 2: Save the photo and comment

[0939] Input: The server receives an upload request.

[0940] Operation: The server stores the photos and comments in a database.

[0941] Output: If the save was successful, the server updates the community feed.

[0942] Specific behavior: The server validates the data and executes an INSERT query to save it to the database.

[0943] Step 3: Update your feed and notify

[0944] Input: The server updates the community feed.

[0945] Operation: The server sends notifications to other users' devices.

[0946] Output: Other users will receive notifications of updates to the feed.

[0947] Specific operation: The server converts the update notification into JSON format and sends it to other user devices as an HTTP PUSH notification.

[0948] This concludes the explanation of the specific processing steps of the overall system program.

[0949] (Application example 1)

[0950] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0951] Currently, there are a wide variety of systems for learning Japanese cuisine recipes, but only a limited number of systems provide practical information useful in the kitchen and support the cooking process. In particular, there are no systems that offer integrated functions such as recipe search, suggestions for alternative ingredients, step-by-step cooking guides, and photo sharing of dishes, as well as the ability to search and view restaurant menus. Therefore, there is a need for a system that allows users to efficiently and easily learn and perform cooking in real-world environments such as Japanese restaurants and cooking classes.

[0952] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0953] In this invention, the server includes means for storing recipe data in a database, means for authenticating users, means for searching for recipes, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, means for searching and browsing restaurant menus, and means for checking how to make dishes step by step in Japanese restaurants or cooking classes, thereby enabling users to efficiently and easily learn and practice cooking in real environments such as Japanese restaurants or cooking classes.

[0954] "Recipe data" refers to information including cooking methods, ingredients, and cooking steps.

[0955] A "database" refers to a system for systematically storing and managing information such as recipe data and user information.

[0956] "User authentication" refers to the process of verifying the information entered (email address and password) when a user logs in to a system and confirming that the user is a legitimate user.

[0957] "Recipe search" refers to the function that extracts recipe information from the database based on entered keywords and provides it to the user.

[0958] "Step-by-step display" refers to a function that breaks down the cooking steps of the selected recipe into order, allowing the user to proceed with the cooking process by checking each step.

[0959] "Substitute material" refers to another material that can perform an equivalent function if a particular material is not available.

[0960] "Means for suggesting alternative materials" refers to a function that suggests suitable alternative materials from a database when a user has difficulty obtaining a particular material.

[0961] "Means for uploading and sharing photos of food" refers to the ability for users to upload photos of food they have cooked to the system and share them with other users.

[0962] "Means for displaying feedback and comments" refers to a function that displays ratings and opinions from other users on the screen so that users can check them.

[0963] "Means for searching and viewing restaurant menus" refers to the function of searching for menus offered at restaurants such as Japanese restaurants and displaying their contents.

[0964] "Cooking classes" refers to facilities and services that provide educational environments and programs for learning how to cook.

[0965] This invention is a system that allows users to effectively learn and practice Japanese cuisine at brick-and-mortar establishments such as Japanese restaurants and cooking schools. This system is primarily composed of a server, user terminals, and a user community. Each element is described in detail below.

[0966] 1. User Registration and Login

[0967] Program Generation

[0968] The server has an API endpoint for user registration and login. The user enters their "username," "email address," and "password" from their device and sends them to the server. The server receives this and stores the user information in a database. When the user logs in, they enter their "email address" and "password," which the server authenticates. After successful authentication, session information is generated and returned to the user.

[0969] Natural language explanations

[0970] In order for a user to use the system, they must first register. The user enters their "user name," "email address," and "password" on their terminal and sends a registration request to the server. The server receives this and saves the user information in a database. The "email address" and "password" entered by the user when logging in are compared with the information in the database for authentication, and if successful, session information is returned to the user. This completes user authentication.

[0971] 2. Search and browse recipes

[0972] Program Generation

[0973] When a user searches for a Japanese recipe, they enter a keyword (e.g., "sushi") into the device's search bar and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and check its details.

[0974] Natural language explanations

[0975] When a user searches for Japanese cuisine, they enter specific keywords into the device's search bar and send a request to the server. The server then extracts the relevant recipe information from the database and sends it back to the device. The user can then select the recipe they are interested in from the displayed list and view detailed information.

[0976] 3. Step-by-step guide

[0977] Program Generation

[0978] The system displays the cooking steps for the recipe selected by the user, step by step. Specifically, the server receives the user's request, retrieves the corresponding step data from the database, and sends it to the device in sequence. The data may include photos and videos, and is displayed in a way that makes it easy for the user to understand.

[0979] Natural language explanations

[0980] A guide is provided to help users check the cooking steps for the recipe they have selected. Upon receiving a user's request, the server retrieves the corresponding step data from the database and sends it to the device in sequence. This allows the user to proceed with the cooking process while checking each step of the cooking procedure.

[0981] 4. Proposal of alternative materials

[0982] Program Generation

[0983] If a user has difficulty obtaining a particular ingredient, the user sends a request for a substitute ingredient from the device to the server, which retrieves the corresponding alternative from the database and returns it to the device. The user can then continue cooking using the suggested alternative.

[0984] Natural language explanations

[0985] If a user feels that a particular ingredient is difficult to obtain, they can send a request to the server for alternative ingredient suggestions. The server will then retrieve the relevant alternative ingredient suggestions from the database and send them back to the device, allowing the user to continue cooking using the suggested alternative ingredients.

[0986] 5. Food photo sharing and community support

[0987] Program Generation

[0988] After the dish is complete, the user takes a photo and uploads it to the server from their device along with their comments. The server stores the photo and comments in a database and updates the community feed, which also displays feedback and comments from other users.

[0989] Natural language explanations

[0990] Users take a photo of their finished dish and upload it to the server along with their comments from their device. The server stores the photo and comments in a database and updates the community feed, displaying feedback and comments from other users and encouraging interaction between users.

[0991] Specific examples

[0992] For example, consider a user who wants to make "sushi." After logging in, the user enters "sushi" in the search bar and presses the search button. The server retrieves recipes related to "sushi" from the database and returns the information to the device. The user selects "sushi" from the displayed list and checks the detailed information. Next, the user proceeds with the cooking process while following the step-by-step guide. If some of the ingredients are unavailable, the user can request alternative ingredients from the server and use the suggested substitute ingredients. Once the dish is complete, the user can upload photos and comments from the device to the server and share them with the community. The user can receive feedback and comments from other users and use them to improve their next cooking experience.

[0993] Prompt Sentence Examples

[0994] "A user launches the app and searches for 'sushi.' The app retrieves recipes related to 'sushi' from the database and displays them to the user. When the user selects a particular recipe, the app displays step-by-step cooking instructions. Also, if a particular ingredient is unavailable, the app suggests alternative ingredients."

[0995] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0996] Step 1:

[0997] User Registration and Login

[0998] Input: Username, Email Address, and Password

[0999] Operation: The device sends input data to the server.

[1000] Data processing: The server validates the data it receives and converts it into a format that can be stored in the database.

[1001] Output: The server saves the user information in the database and returns a registration success message to the terminal.

[1002] Step 2:

[1003] User authentication

[1004] Input: User's email address and password

[1005] How it works: The device sends authentication data to the server.

[1006] Data calculation: The server compares the data it receives with the information in the database.

[1007] Output: If authentication is successful, generate session information and return it to the terminal. If authentication fails, return an error message.

[1008] Step 3:

[1009] Search and browse recipes

[1010] Input: User search keyword (e.g. "sushi")

[1011] How it works: The device sends a search request to the server.

[1012] Data processing: The server searches the database based on keywords and extracts relevant recipe information.

[1013] Output: Returns search results to the device and displays a list of recipes to the user.

[1014] Step 4:

[1015] Step-by-step guide

[1016] Input: Recipe ID selected by the user

[1017] How it works: The device sends a request to the server.

[1018] Data calculation: The server retrieves the step data corresponding to the recipe ID from the database.

[1019] Output: Sends step data (including photos and videos) to the device sequentially and displays them to the user.

[1020] Step 5:

[1021] Proposal of alternative materials

[1022] Input: User request for specific material alternatives

[1023] How it works: The device sends a request to the server.

[1024] Data calculation: The server searches and extracts the relevant alternative materials from the database.

[1025] Output: Sends the proposed alternative material information to the terminal.

[1026] Step 6:

[1027] Food photo sharing and community support

[1028] Input: User-taken food photos and comments

[1029] What happens: Your device uploads the photo and comments to the server.

[1030] Data processing: The server receives the photos and comments and stores them in a database.

[1031] Output: Update the community feed and display the new information on other users' devices.

[1032] Examples of concrete examples and prompts

[1033] For example, if a user wants to make "sushi," the process would follow these steps:

[1034] Step 1: The user enters their username, email address, and password on their device and sends them to the server. The server receives them and stores them in a database.

[1035] Step 2: The user enters their email address and password for authentication. The server compares the information with the database and returns session information if authentication is successful.

[1036] Step 3: The user types "sushi" into the search bar and sends a search request to the server, which searches its database for recipes related to "sushi" and returns them to the device.

[1037] Step 4: The user requests the selected recipe from the server and receives step-by-step cooking instructions.

[1038] Step 5: If a particular ingredient is not available, a request for an alternative ingredient is sent to the server, which then suggests an alternative ingredient.

[1039] Step 6: After the dish is completed, the user uploads the photo and comment to the server and shares it with the community. They receive feedback and comments from other users.

[1040] Example prompt sentence:

[1041] "A user launches the app and searches for 'sushi.' The app retrieves recipes related to 'sushi' from the database and displays them to the user. When the user selects a particular recipe, the app displays step-by-step cooking instructions. Also, if a particular ingredient is unavailable, the app suggests alternative ingredients."

[1042] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1043] The system of the present invention includes means for storing recipe data in a database, means for user authentication, means for recipe search, means for displaying step-by-step recipe instructions, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, and an emotion engine for recognizing user emotions.

[1044] User Registration and Login

[1045] In order for a user to use the system, they must first register. The user enters their "user name," "email address," and "password" on their device and sends a registration request to the server. The server receives this and stores the user information in a database. Once registration is complete, the user enters their "email address" and "password" on the login screen and is authenticated. The server compares the information with that in the database and, if successful, returns session information to the user.

[1046] Search and browse recipes

[1047] When a user searches for a Japanese recipe, they enter a keyword (e.g., "sushi") into the device's search bar and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and check its details.

[1048] Step-by-step guide

[1049] The user can check each cooking step on the details screen of the selected recipe. The server receives a request from the device and retrieves the corresponding step data (including photos and videos) from the database and sends them to the device in sequence. The user proceeds with the cooking process by checking each step displayed on the screen.

[1050] Proposal of alternative materials

[1051] If a user has difficulty obtaining a particular ingredient, they need to be offered an alternative. In this case, the device sends a request for an alternative ingredient to the server. The server retrieves the relevant alternative from the database and returns it to the device. The user can then continue cooking using the alternative provided.

[1052] Food photo sharing and community support

[1053] After the dish is complete, the user takes a photo and uploads it to the server along with their comments. The server stores the photo and comments in a database and updates the community feed. This update is then reflected on other users' devices, allowing them to receive feedback and comments from other users. This promotes interaction and support among users.

[1054] Introducing the Emotion Engine

[1055] The app has an emotion engine that recognizes the user's emotions, and can analyze their facial expressions and text comments to suggest suitable recipes and substitute ingredients.

[1056] For example, consider a case where a user wants to make "sushi." After logging in, the user enters "sushi" in the search bar and presses the search button. The server extracts recipes related to "sushi" from the database and returns the information to the device. The user selects "sushi" from the displayed list and checks the detailed information.

[1057] Next, you cook using a step-by-step guide. If you don't have some ingredients, you can request a substitute from the server and use the suggested alternative. Once your dish is complete, you can upload a photo and comments from your device to the server and share them with the community. You can receive feedback and comments from other users to improve your next cooking experience.

[1058] Furthermore, the emotion engine can recognize emotions from the user's facial expressions and comments, suggesting more difficult recipes if the user is having fun, and easier recipes if the user is feeling down. This type of customized feedback is provided by the emotion engine.

[1059] In this way, the system of the present invention supports users in the process of learning and enjoying Japanese cuisine and promotes mutual interaction through the community. Furthermore, the introduction of an emotion engine provides more personalized services and improves user satisfaction.

[1060] The processing flow will be explained below.

[1061] Modes for implementing the invention combining emotion engines

[1062] User Registration and Login

[1063] Step 1:

[1064] User: Enter your "Username", "Email address", and "Password" in the new registration form and press the registration button.

[1065] Step 2:

[1066] Terminal: Converts the input information into JSON format and sends an API request to the server.

[1067] Step 3:

[1068] Server: Receives the API request and validates the input information.

[1069] Step 4:

[1070] Server: After verification, save the new user information to the database.

[1071] Step 5:

[1072] Server: Returns a message to the terminal indicating that saving is complete.

[1073] Step 6:

[1074] Terminal: Display a save complete message to the user.

[1075] Step 7:

[1076] User: Enter your email address and password in the login form and click the login button.

[1077] Step 8:

[1078] Terminal: Converts login information into JSON format and sends an API request to the server.

[1079] Step 9:

[1080] Server: Receives the API request, retrieves the relevant user information from the database, and collates it.

[1081] Step 10:

[1082] Server: If authentication is successful, it creates a user session and returns a session token to the device.

[1083] Step 11:

[1084] Terminal: Receives the session token and logs the user in.

[1085] Search and browse recipes

[1086] Step 1:

[1087] User: Enter a keyword (e.g., "sushi") in the search bar and press the search button.

[1088] Step 2:

[1089] Terminal: Converts search keywords into JSON format and sends an API request to the server.

[1090] Step 3:

[1091] Server: Receives the API request and searches the database for the corresponding recipe information.

[1092] Step 4:

[1093] Server: Converts the relevant recipe information into JSON format and returns it to the device.

[1094] Step 5:

[1095] Terminal: Displays the received recipe information to the user.

[1096] View detailed recipes and step-by-step guides

[1097] Step 1:

[1098] User: Select and click on the recipe you are interested in from the list of recipes displayed.

[1099] Step 2:

[1100] Terminal: Convert the selected recipe ID into JSON format and send an API request to the server.

[1101] Step 3:

[1102] Server: Receives the API request and retrieves the corresponding recipe details from the database.

[1103] Step 4:

[1104] Server: Converts detailed information into JSON format and returns it to the terminal.

[1105] Step 5:

[1106] Terminal: Displays the received details to the user.

[1107] Step 6:

[1108] User: On the recipe details screen, follow the step-by-step guide as you cook.

[1109] Step 7:

[1110] Terminal: Requests next step data from the server based on the user's progress.

[1111] Step 8:

[1112] Server: Retrieves the next step data from the database and returns it to the terminal.

[1113] Step 9:

[1114] Terminal: Display step data to the user.

[1115] Proposal of alternative materials

[1116] Step 1:

[1117] User: Clicks a button to request an alternative material suggestion for a particular material.

[1118] Step 2:

[1119] Terminal: Converts material information into JSON format and sends an API request to the server to request a substitute material.

[1120] Step 3:

[1121] Server: Receives the request and retrieves the relevant information on alternative materials from the database.

[1122] Step 4:

[1123] Server: Converts the alternative material information into JSON format and returns it to the terminal.

[1124] Step 5:

[1125] Terminal: Displays the received alternative material information to the user.

[1126] Food photo sharing and community support

[1127] Step 1:

[1128] User: Take a photo of the dish and enter it along with a comment on the upload screen.

[1129] Step 2:

[1130] Device: Convert the photo and comment into JSON format and send an API request to the server to upload.

[1131] Step 3:

[1132] Server: Receives the request and stores the photos and comments in a database.

[1133] Community Feed Updates

[1134] Step 1:

[1135] Server: Update the community feed every time a new post is made.

[1136] Step 2:

[1137] Device: Receives updated feed information and displays it on the user's screen.

[1138] Step 3:

[1139] Users: View and interact with other users' feedback and comments.

[1140] Introducing the Emotion Engine

[1141] Step 1:

[1142] User: Enter facial expressions or text comments while using the system.

[1143] Step 2:

[1144] Terminal: Sends the user's facial expression data or text comments to the emotion engine.

[1145] Step 3:

[1146] Emotion Engine: Analyzes the received data and identifies the user's emotions.

[1147] Step 4:

[1148] Server: Based on the analysis results from the emotion engine, it suggests suitable recipes and alternative ingredients.

[1149] Step 5:

[1150] On the device: Display suggestions to the user.

[1151] Step 6:

[1152] User: Proceed with cooking using suggested recipes and alternative ingredients.

[1153] For example, if a user wants to make "sushi," they enter "sushi" in the search bar and press the search button. The server retrieves recipes related to "sushi" from the database and returns them to the device. The user selects "sushi" from the displayed list and checks the details. If a substitute ingredient is needed during the cooking process, a request is sent to the server and the suggested ingredient is used. Once the dish is complete, the user can upload a photo and comment and receive feedback from the community.

[1154] Furthermore, the emotion engine recognizes the user's emotions from their facial expressions and comments, and suggests more difficult recipes if they are having fun, and easier recipes if they are feeling down, providing this kind of customized feedback.

[1155] Example 2

[1156] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1157] Previous recipe search systems had issues such as a lack of specific instructions required for users to cook a dish and a lack of suggestions for alternative ingredients. They also lacked functionality for users to share the cooking process and results with other users, and were unable to customize recipes based on the user's feelings. This led to issues such as low user satisfaction and a low rate of continued use of the system.

[1158] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1159] In this invention, the server includes means for storing recipe data in a database, means for authenticating users, means for searching for recipes, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, means for receiving requests from the terminal and retrieving corresponding data from the database, and an emotion engine for identifying and analyzing user emotions. This allows users to efficiently obtain the information necessary for cooking Japanese cuisine and enjoy cooking while interacting with other users. Furthermore, by utilizing the emotion engine, it is possible to suggest optimal recipes and alternative ingredients based on the user's emotions, thereby improving user satisfaction.

[1160] "Recipe data" refers to data that includes information such as cooking steps, necessary ingredients, quantities, cooking methods, etc.

[1161] A "database" is a system or device for efficiently storing, managing, and searching large amounts of data.

[1162] "User authentication" is the process of verifying that a user is authorized to access a system.

[1163] "Recipe search" is the process of extracting desired recipe data from a database based on conditions specified by the user.

[1164] "Step-by-step" refers to showing the recipe cooking steps step by step.

[1165] "Alternative materials" are substitute materials that can be used when a particular material is unavailable or according to user preference.

[1166] "Uploading" is the act of a user sending data from their own device to a server.

[1167] "Sharing" refers to making data uploaded by a user available for viewing and use by other users.

[1168] "Feedback" refers to reaction information such as comments and ratings provided by other users.

[1169] A "request" refers to a user requesting specific information or service from a server.

[1170] An "emotion engine" is a system or algorithm that analyzes emotions from a user's facial expressions and text and makes appropriate suggestions.

[1171] The system of the present invention is a comprehensive system that stores recipe data in a database, authenticates users, and provides multiple functions, such as recipe search, step-by-step recipe instructions, suggests alternative ingredients, and allows users to upload and share photos of their dishes. It also includes a sentiment engine that displays feedback and comments from other users and identifies and analyzes user sentiment.

[1172] System configuration

[1173] Hardware and Software Used

[1174] This system is primarily built using servers, databases, and user devices (smartphones and PCs). The software used includes a web application framework, a database management system, image analysis software, and sentiment analysis algorithms.

[1175] Servers: Includes web servers, database servers, and AI engine servers.

[1176] Database: Use SQL database or NoSQL database.

[1177] User devices: Smartphones, PCs, tablets, etc.

[1178] Software: Web application frameworks (e.g., Django, Ruby on Rails), database management systems (e.g., MySQL, PostgreSQL), image analysis software (e.g., OpenCV), sentiment analysis algorithms (e.g., IBM Watson, Google Cloud Natural Language API)

[1179] Explanation of program processing

[1180] User Registration and Login

[1181] When a user uses the system, they must first register. The user enters their "user name," "email address," and "password" from their device and sends a registration request to the server. The server receives this and stores the user information in a database. Once registration is complete, the user enters their "email address" and "password" on the login screen and is authenticated. The server compares the information with that in the database and, if successful, returns session information to the user.

[1182] Examples:

[1183] Prompt statement:

[1184] Enter your email address:

[1185] Please enter your password:

[1186] Enter your username:

[1187] Search and browse recipes

[1188] When a user searches for a recipe, they enter a keyword (e.g., "sushi") into the search bar on their device and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and view its details.

[1189] Examples:

[1190] Prompt statement:

[1191] Enter the name of the dish you want to search for recipes for:

[1192] Example: Sushi

[1193] Step-by-step guide

[1194] The user can check each cooking step on the details screen of the selected recipe. The server receives a request from the device and retrieves the corresponding step data (including photos and videos) from the database and sends them to the device in sequence. The user proceeds with the cooking process by checking each step displayed on the screen.

[1195] Proposal of alternative materials

[1196] If a user has difficulty obtaining a particular ingredient, they need to be offered an alternative. In this case, the device sends a request for an alternative ingredient to the server. The server retrieves the relevant alternative from the database and returns it to the device. The user can then continue cooking using the alternative provided.

[1197] Food photo sharing and community support

[1198] After the dish is complete, the user takes a photo and uploads it to the server along with their comments. The server stores the photo and comments in a database and updates the community feed. This update is then reflected on other users' devices, allowing them to receive feedback and comments from other users. This promotes interaction and support among users.

[1199] Introducing the Emotion Engine

[1200] The system has an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's emotions from their facial expressions and text comments, and can suggest appropriate recipes and alternative ingredients. Specifically, if a user comments, "I'm in a good mood today!", the system will suggest more difficult recipes. Conversely, if a user comments, "I'm tired today," the system will suggest easier recipes.

[1201] Specific examples

[1202] Here is a specific use case where the user wants to make sushi:

[1203] 1. User registration and login: Enter your username, email address, and password to register in the system, then log in to use the system.

[1204] 2. Recipe search: Enter "sushi" into the search bar on the device. The server retrieves recipes related to "sushi" from the database and returns the information to the device.

[1205] 3. Step-by-step guide: Users choose a sushi recipe and follow the step-by-step cooking instructions as they go through the process.

[1206] 4. Suggestion of alternative materials: If some of the required materials are not on hand, a request for alternative materials is made to the system, and the server proposes alternatives.

[1207] 5. Share your cooking photos: Once you've finished cooking, upload photos and comments to share with other users and get feedback.

[1208] 6. Use of emotion engine: Recognize emotions from the user's facial expressions and comments and suggest appropriate recipes.

[1209] In this way, the system of the present invention comprehensively supports users in the process of learning and enjoying cooking, and also promotes interaction with other users. Furthermore, the introduction of an emotion engine makes it possible to provide personalized services according to the user's emotions, thereby improving user satisfaction.

[1210] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1211] User Registration and Login

[1212] Step 1:

[1213] Input: The user enters their "username," "email address," and "password" on the device.

[1214] Processing: The terminal sends a registration request containing this information to the server.

[1215] Output: None.

[1216] Specific operation: When the user clicks the "New Registration" button, the terminal creates and sends a request to send the entered information to the server.

[1217] Step 2:

[1218] Input: The server receives a registration request from the terminal.

[1219] Processing: The server stores the received user information in a database.

[1220] Output: Registration successful message.

[1221] Specific operation: After the server saves the user information in the database, it returns a registration success message to the terminal.

[1222] Step 3:

[1223] Input: The user enters their email address and password on the login screen.

[1224] Processing: The terminal sends a login request containing this information to the server.

[1225] Output: None.

[1226] Specific operation: When you click the "Login" button on the login screen, the device creates a login request based on the information entered and sends it to the server.

[1227] Step 4:

[1228] Input: The server receives a login request.

[1229] Processing: The server checks the user information against the database and performs authentication.

[1230] Output: If authentication is successful, session information is generated and returned to the terminal.

[1231] What happens: The server checks the user's email address and password against records in its database, and if authentication is successful, generates and returns session information.

[1232] Search and browse recipes

[1233] Step 1:

[1234] Input: The user enters a keyword (e.g., "sushi") into the device's search bar.

[1235] Processing: The terminal sends a search request to the server.

[1236] Output: None.

[1237] Specific operation: When the user clicks the "Search" button, the device creates a search request based on the entered keywords and sends it to the server.

[1238] Step 2:

[1239] Input: The server receives a search request.

[1240] Processing: The server extracts the relevant recipe information from the database.

[1241] Output: Relevant recipe information.

[1242] Specific operation: The server searches the recipe database and extracts recipe information that matches the keywords in list form.

[1243] Step 3:

[1244] Input: The server sends the extracted recipe information to the terminal.

[1245] Processing: The device displays the received recipe information to the user.

[1246] Output: A list of displayed recipes.

[1247] Specific operation: The terminal displays the recipe information received from the server on the screen.

[1248] Step 4:

[1249] Input: The user selects the recipe of interest from the displayed list of recipes.

[1250] Processing: The device sends a request to the server to display the details of the selected recipe.

[1251] Output: None.

[1252] Specific operation: When the user clicks on the selected recipe, the device requests detailed information about that recipe from the server.

[1253] Step-by-step guide

[1254] Step 1:

[1255] Input: User requests cooking instructions on the recipe details screen.

[1256] Processing: The terminal sends a cooking instructions request to the server.

[1257] Output: None.

[1258] Specific operation: When the user clicks the "Start Cooking" button, the terminal creates a cooking procedure request and sends it to the server.

[1259] Step 2:

[1260] Input: The server receives a cooking instructions request.

[1261] Processing: The server retrieves cooking recipe data from the database.

[1262] Output: Cooking instructions data.

[1263] Specific operation: The server retrieves the cooking procedure data for the corresponding recipe from the database and returns it to the terminal along with photo and video information.

[1264] Step 3:

[1265] Input: The server sends the acquired cooking procedure data to the terminal.

[1266] Processing: The terminal displays the received procedure data to the user in sequence.

[1267] Output: Displayed cooking instructions.

[1268] Specific operation: The device receives cooking instructions from the server and displays them on the screen step by step. The user proceeds with the cooking process by checking each step one by one.

[1269] Proposal of alternative materials

[1270] Step 1:

[1271] Input: If a user does not have a specific ingredient available, they can submit a request for a substitute ingredient from their device.

[1272] Processing: The terminal sends a substitute material request to the server.

[1273] Output: None.

[1274] Specific operation: When the user clicks the "Search for alternative materials" button, the terminal creates a request for alternative materials and sends it to the server.

[1275] Step 2:

[1276] Input: The server receives a substitute material request.

[1277] Processing: The server retrieves the relevant alternative material information from the database.

[1278] Output: Alternative material information.

[1279] Specific operation: The server retrieves alternatives to the requested ingredients from the database (e.g., tuna as an alternative to salmon) and returns them to the terminal.

[1280] Step 3:

[1281] Input: The server sends the acquired alternative material information to the terminal.

[1282] Processing: The terminal displays the received alternative material information to the user.

[1283] Output: Displayed alternative material information.

[1284] Specific operation: The device displays the information on alternative ingredients received from the server on the screen, and the user can use it as a reference to continue cooking.

[1285] Food photo sharing and community support

[1286] Step 1:

[1287] Input: The user takes a photo of the finished dish and enters a comment.

[1288] Processing: The device sends an upload request containing the photo and comment to the server.

[1289] Output: None.

[1290] Specific operation: When the user clicks the "Upload Photo" button, the device will send the taken photo and any comments entered to the server.

[1291] Step 2:

[1292] Input: The server receives an upload request.

[1293] Processing: The server stores the photos and comments in a database.

[1294] Output: Save complete message.

[1295] Specific operation: The server saves the received photos and comments in a database and returns a save completion message to the terminal.

[1296] Step 3:

[1297] Input: The server updates the community feed based on the stored photos and comments.

[1298] Processing: The server creates new feed data and sends it to other user devices.

[1299] Output: The updated community feed.

[1300] Specific operation: The server updates the community feed based on the user's upload and notifies other users of the data.

[1301] Introducing the Emotion Engine

[1302] Step 1:

[1303] Input: User facial expressions and text comments.

[1304] Processing: The server collects emotion data from users' facial expressions and text comments.

[1305] Output: Emotion data.

[1306] How it works: When a user types a comment or uploads a photo, the system sends a request to the emotion engine to analyze the user's facial expressions and comments.

[1307] Step 2:

[1308] Input: Emotion data.

[1309] Processing: The emotion engine analyzes the collected data and identifies the user's emotional state.

[1310] Output: Sentiment analysis results.

[1311] Specific behavior: The emotion engine analyzes the collected data and identifies emotional states, such as "happy" or "tired."

[1312] Step 3:

[1313] Input: Sentiment analysis results.

[1314] Processing: The server suggests appropriate recipes and alternative ingredients based on the sentiment analysis results.

[1315] Output: Sentiment-based suggestion information.

[1316] Specific operation: The server receives the sentiment analysis results and suggests difficult or easy recipes to the user based on them.

[1317] In this way, the system of the present invention comprehensively supports users in the process of learning and enjoying cooking, and also promotes interaction with other users. Furthermore, the introduction of an emotion engine makes it possible to provide personalized services according to the user's emotions, thereby improving user satisfaction.

[1318] (Application example 2)

[1319] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1320] Conventional online recipe systems provide basic functionality for users to search for recipes and follow the instructions to cook, but this alone is insufficient. With the recent proliferation of food delivery services, there is a demand for systems that comprehensively support the process from ordering food to receiving it and enjoying it. Furthermore, there is a need for systems that can appropriately suggest alternative ingredients when ingredients are out of stock, and make personalized suggestions based on the user's emotions. Conventional systems lack such advanced functionality, making improving the user experience a challenge.

[1321] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1322] In this invention, the server includes means for storing recipe data in a database, means for user authentication, means for searching for recipes, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, means for providing step-by-step guidance from receiving to enjoying the ordered dish, means for suggesting alternative ingredients if an ingredient is out of stock at the time of ordering, and means for recognizing user emotions and providing personalized service. This allows users to enjoy the cooking process with guidance even after ordering and receiving the dish. Furthermore, by providing appropriate alternative suggestions even if an ingredient is out of stock, and by making personalized suggestions based on the user's emotions, a significant improvement in the user experience can be expected.

[1323] "Recipe data" is a collection of information stored in a database, including cooking instructions, necessary ingredients, and steps.

[1324] "User authentication" is the process of verifying the identification information (user name, password, etc.) of a system user and verifying that the user is a legitimate user.

[1325] "Recipe Search" is a system feature that allows users to search for information about specific dishes or ingredients.

[1326] "Step-by-step display" is a system function that guides users through each step of a recipe, checking them one by one.

[1327] "Alternative material suggestions" is a system function that suggests alternative materials when a user has difficulty obtaining a particular material.

[1328] "Means to upload and share photos of food" refers to a function that allows users to take photos of the food they have made and share them with other users on the system.

[1329] "Show feedback and comments" is a system feature that displays opinions and comments from other users.

[1330] "Step-by-step guidance" is a system function that provides detailed explanations and guidance for users to enjoy the food after the ordered food arrives.

[1331] An "emotion recognition engine" is a system function that analyzes emotions from a user's facial expressions and text comments and provides personalized services based on that.

[1332] "Personalized service provision" refers to the system's ability to provide users with information and suggestions that are best suited to them based on their specific interests and emotions.

[1333] The following system configuration is used as an embodiment of the present invention.

[1334] System Overview

[1335] The system includes a database, user authentication mechanism, recipe search function, step-by-step guidance function, ingredient substitution suggestion function, community function, emotion recognition engine, and personalized service provision function, providing comprehensive support that allows users to enjoy the cooking process from ordering to receiving their food.

[1336] Hardware and software used

[1337] Hardware: Smartphones, smart glasses, head-mounted displays, servers

[1338] Software: Flask (a Python web framework), SQLAlchemy (for database operations), emotion recognition engine (facial expression analysis and text analysis)

[1339] Details of each function

[1340] Building a database

[1341] The server stores recipe data in a database, which includes the name of the dish, the ingredients needed, the cooking instructions, and photos or videos. The database is operated using SQLAlchemy.

[1342] User authentication

[1343] In order for a user to use the system, they must register and log in. The authentication information (username, email address, password) is verified by the server, and session information is generated after authentication.

[1344] Recipe Search

[1345] The user enters keywords into the search bar and sends a recipe search request to the server. The server searches the database for the relevant recipe and returns it to the device. For example, it is possible to search for the keyword "sushi."

[1346] Step-by-step guide

[1347] The recipe details screen for a user's selected recipe displays step-by-step cooking instructions, each supplemented with photos and videos to guide the user through the cooking process.

[1348] Proposal of alternative materials

[1349] If some of the ingredients are unavailable, the user can request an alternative, and the server will retrieve the appropriate alternative from the database and return it to the user's device.

[1350] Community Features

[1351] Once the dish is complete, the user can take a photo and upload it along with a comment, which the server stores in a database and displays to other users.

[1352] Emotion Recognition Engine

[1353] Emotions are analyzed from the user's facial expressions and text comments, allowing the server to adjust the difficulty of recipes and make personalized suggestions based on the user's emotional state.

[1354] Specific examples

[1355] For example, if a user wants to make "sushi," they follow these steps: They log in to the application and type "sushi" into the search bar to search for related recipes. They select a sushi recipe that appears and follow a step-by-step guide to cook it. If some of the ingredients are unavailable, they request alternative ingredients and accept suggestions from the server. Once cooking is complete, they take a photo of the dish and share it with the community along with a comment. Furthermore, an emotion recognition engine analyzes the user's facial expressions and comments to see if they are enjoying the dish, and can suggest a recipe with an adjusted difficulty for next time.

[1356] Prompt Sentence Examples

[1357] "Just search for the name of the recipe you're interested in and get detailed step-by-step instructions on how to make it. For example, 'Sushi' or 'Curry'."

[1358] In this way, the system of the present invention can assist users in the process of enjoying cooking and improve the user experience.

[1359] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1360] Step 1:

[1361] A user accesses the system and registers.

[1362] Input: The user enters their "username," "email address," and "password" from their device and sends a registration request to the server.

[1363] Processing: The server stores the received information in a database.

[1364] Output: The server returns a registration completion message to the user terminal.

[1365] Specific operation: A user opens the application and accesses the new registration screen. They enter the required information and press the "Register" button. The server receives this and saves the user information to the database using an SQL query.

[1366] Step 2:

[1367] A user logs in with an existing account.

[1368] Input: The user enters their email address and password on the device and sends a login request to the server.

[1369] Processing: The server checks the information in its database and performs authentication.

[1370] Output: If the authentication is successful, the server generates session information and returns it to the user terminal.

[1371] Specific operation: The user goes to the login screen, enters the required authentication information, and presses the "Login" button. The server receives this and compares it with the information in the database. If the comparison is successful, the server generates session information and returns it to the user's device.

[1372] Step 3:

[1373] A user searches for a recipe.

[1374] Input: The user enters a keyword (e.g., "sushi") into the search bar of the device and sends a search request to the server.

[1375] Processing: The server extracts the relevant recipe information from the database.

[1376] Output: The server returns a list of search results to the user's device.

[1377] How it works: The user enters keywords into the search bar and presses the "Search" button. The server executes an SQL query to retrieve related recipe information from the database and returns the results to the user's device.

[1378] Step 4:

[1379] The user reviews the recipe details and follows the step-by-step instructions.

[1380] Input: Request details of the recipe selected by the user.

[1381] Processing: The server retrieves data for each step (including photos and videos) from the database.

[1382] Output: The server sends step-by-step instructions to the user terminal sequentially.

[1383] Specific operation: The user selects a recipe of interest from the displayed list of recipes and presses the details button. The server retrieves the corresponding step data from the database and sequentially sends it to the user's terminal.

[1384] Step 5:

[1385] If materials are unavailable, suggest alternative materials.

[1386] Input: The user submits a request for a substitute material from the terminal.

[1387] Processing: The server retrieves data on the corresponding alternative materials from the database.

[1388] Output: The server returns the alternative material suggestions to the user terminal.

[1389] Specific operation: If a user does not have a specific ingredient, they request an alternative to that ingredient. The server executes an SQL query to retrieve the alternative ingredient information from the database and returns it to the user's device.

[1390] Step 6:

[1391] Users upload photos of their dishes and share them with the community.

[1392] Input: After the user has finished cooking, they take a photo and send an upload request to the server along with their comments.

[1393] Processing: The server stores the received photos and comments in a database and updates the community feed.

[1394] Output: The server reflects the updated feed information to the user's terminal.

[1395] What it does: Users use the application's "Photo Upload" feature to submit photos of their finished dishes along with their comments. The server receives these, stores them in a database, and updates the community feed, which also displays feedback and comments from other users.

[1396] Step 7:

[1397] Recognize user emotions using an emotion engine.

[1398] Input: The user's facial expression data and text comments are taken as input.

[1399] Processing: The emotion engine analyzes the input data and determines the user's emotional state.

[1400] Output: Display recipe difficulty adjustment and personalized serving suggestions based on emotions.

[1401] Specific operation: The user's facial expression is captured by a camera or a comment is entered. The emotion engine analyzes this data to determine their emotional state (happy, sad, excited, etc.) and suggests the most appropriate service or recipe based on the results. For example, if the user looks happy, it will suggest a more difficult recipe, and conversely, if the user looks depressed, it will suggest an easy recipe.

[1402] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1403] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1404] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1405] [Third embodiment]

[1406] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1407] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1408] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1409] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1410] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1411] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1412] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1413] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1414] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1415] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1416] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1417] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1418] The system of the present invention is mainly composed of a server, user terminals, and a user community. Each element will be explained separately below.

[1419] User Registration and Login

[1420] In order for a user to use the system, they must first register. At this time, the user enters their "user name," "email address," and "password" from their device and sends a registration request to the server. The server receives this and stores the user information in a database. Once registration is complete, the user enters their "email address" and "password" on the login screen and is authenticated. The server compares the information with that in the database and, if successful, returns session information to the user.

[1421] Search and browse recipes

[1422] When a user searches for a Japanese recipe, they enter a keyword (e.g., "sushi") into the device's search bar and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and check its details.

[1423] Step-by-step guide

[1424] For the recipe selected by the user, each cooking step can be checked on the details screen. Upon receiving a request from the device, the server retrieves the corresponding step data (including photos and videos) from the database and sends them to the device in sequence. The user proceeds with the cooking process by checking each step displayed on the screen.

[1425] Proposal of alternative materials

[1426] If a user has difficulty obtaining a particular ingredient, they need to be offered an alternative. In this case, the device sends a request for an alternative ingredient to the server. The server retrieves the relevant alternative from the database and returns it to the device. The user can then continue cooking using the alternative provided.

[1427] Food photo sharing and community support

[1428] After the dish is complete, the user takes a photo and uploads it to the server along with their comments. The server stores the photo and comments in a database and updates the community feed. This update is then reflected on other users' devices, allowing them to receive feedback and comments from other users. This promotes interaction and support among users.

[1429] Specific examples

[1430] For example, consider a case where a user wants to make "sushi." After logging in, the user enters "sushi" in the search bar and presses the search button. The server extracts recipes related to "sushi" from the database and returns the information to the device. The user selects "sushi" from the displayed list and checks the detailed information.

[1431] Next, you cook using a step-by-step guide. If you don't have some ingredients, you can request a substitute from the server and use the suggested alternative. Once your dish is complete, you can upload a photo and comments from your device to the server and share them with the community. You can receive feedback and comments from other users to improve your next cooking experience.

[1432] In this way, the system of the present invention supports users in the process of learning and enjoying Japanese cuisine, and promotes mutual interaction through the community.

[1433] The processing flow will be explained below.

[1434] User Registration and Login

[1435] User Registration

[1436] Step 1:

[1437] User: Enter your "Username", "Email address", and "Password" in the new registration form and press the registration button.

[1438] Step 2:

[1439] Device: Converts registration information into JSON format and sends an API request to the server.

[1440] Step 3:

[1441] Server: Receives the API request and checks the format and content of the input data.

[1442] Step 4:

[1443] Server: Save the new user information to the database.

[1444] Step 5:

[1445] Server: If the save is successful, it returns a success message to the terminal.

[1446] Step 6:

[1447] Terminal: Receives the success message and displays it to the user.

[1448] Log in

[1449] Step 1:

[1450] User: Enter your email address and password in the login form and click the login button.

[1451] Step 2:

[1452] Terminal: Converts login information into JSON format and sends an API request to the server.

[1453] Step 3:

[1454] Server: Receives the API request and retrieves the corresponding user information from the database.

[1455] Step 4:

[1456] Server: Compares the acquired user information with the entered information, and performs authentication if they match.

[1457] Step 5:

[1458] Server: If authentication is successful, it generates session information and returns it to the terminal.

[1459] Step 6:

[1460] Terminal: Receives session information and logs the user in.

[1461] Search and browse recipes

[1462] Recipe Search

[1463] Step 1:

[1464] User: Enter a keyword (e.g., "sushi") in the search bar and press the search button.

[1465] Step 2:

[1466] Terminal: Converts search keywords into JSON format and sends an API request to the server.

[1467] Step 3:

[1468] Server: Receives the API request and searches the database for the corresponding recipe information.

[1469] Step 4:

[1470] Server: Converts the relevant recipe information into JSON format and returns it to the device.

[1471] Step 5:

[1472] Terminal: Displays the received recipe information to the user.

[1473] View recipe details

[1474] Step 1:

[1475] User: Select and click on the recipe you are interested in from the list of recipes displayed.

[1476] Step 2:

[1477] Terminal: Convert the selected recipe ID into JSON format and send an API request to the server.

[1478] Step 3:

[1479] Server: Receives the API request and retrieves the corresponding recipe details from the database.

[1480] Step 4:

[1481] Server: Converts detailed information into JSON format and returns it to the terminal.

[1482] Step 5:

[1483] Terminal: Displays the received details to the user.

[1484] Step-by-step guide

[1485] View Guide

[1486] Step 1:

[1487] User: On a recipe detail screen, clicks a link or button that displays a step-by-step guide.

[1488] Step 2:

[1489] Terminal: Convert the recipe ID into JSON format and send an API request for step data to the server.

[1490] Step 3:

[1491] Server: Receives the API request and retrieves the corresponding step data from the database.

[1492] Step 4:

[1493] Server: Converts step data (including photos and videos) into JSON format and returns it to the device.

[1494] Step 5:

[1495] Terminal: Displays the received step data to the user.

[1496] Proposal of alternative materials

[1497] Request an alternative material

[1498] Step 1:

[1499] User: Clicks a button to request an alternative material suggestion for a particular material.

[1500] Step 2:

[1501] Terminal: Converts material information into JSON format and sends an API request to the server to request a substitute material.

[1502] Step 3:

[1503] Server: Receives the request and retrieves the relevant information on alternative materials from the database.

[1504] Step 4:

[1505] Server: Converts the alternative material information into JSON format and returns it to the terminal.

[1506] Step 5:

[1507] Terminal: Displays the received alternative material information to the user.

[1508] Food photo sharing and community support

[1509] Upload a photo

[1510] Step 1:

[1511] User: Take a photo of the dish and enter it along with a comment on the upload screen.

[1512] Step 2:

[1513] Device: Convert the photo and comment into JSON format and send an API request to the server to upload.

[1514] Step 3:

[1515] Server: Receives the request and stores the photos and comments in a database.

[1516] Community Feed Updates

[1517] Step 1:

[1518] Server: Update the community feed every time a new post is made.

[1519] Step 2:

[1520] Device: Receives updated feed information and displays it on the user's screen.

[1521] Step 3:

[1522] Users: View and interact with other users' feedback and comments.

[1523] The above are the specific processing steps for carrying out the present invention.

[1524] Example 1

[1525] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1526] With conventional recipe search systems, it was difficult for users to find the recipe they were looking for, and they lacked a function to suggest alternative ingredients when ingredients were unavailable, which often hindered the progress of cooking.In addition, interaction between users was limited, and they were unable to receive feedback or comments from other users, which made it difficult to improve their cooking skills or discover new recipes.

[1527] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1528] In this invention, the server includes means for storing user information in a database when a user registers, means for authenticating the user, means for searching and extracting recipe data from the database, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for the user to upload and share photos of the dish, means for displaying feedback and comments from other users, and means for displaying alternative ingredients on the user's terminal if the user cannot obtain a particular ingredient. This allows users to easily search and view desired recipes and receive suggestions for alternative ingredients even if the ingredients are unavailable. Furthermore, sharing photos of dishes promotes interaction within the community, and by receiving feedback and comments from other users, users can improve their cooking skills.

[1529] "User registration" refers to the process by which a user provides their own information to the system and is authenticated as a user.

[1530] "Storing in a database" refers to the persistent storage of digital information so that it can be accessed at a later time.

[1531] "User authentication" is the process by which a system verifies a user's identity and grants appropriate access privileges.

[1532] "Searching and extracting recipe data from a database" means searching for recipe information in a database based on specific conditions and extracting the corresponding data.

[1533] "Step-by-step" means showing the user steps or operations in a sequential, step-by-step manner.

[1534] "Suggesting alternative materials" means suggesting alternative materials that provide the same effect if the user cannot obtain a particular material.

[1535] "Users upload and share photos of their dishes" means that users send photos of their own dishes to the system and share the information with other users.

[1536] "Displaying feedback and comments from other users" means that the system collects opinions and reactions from other users and presents them in a visible form to the user.

[1537] "Displaying alternative materials on the user's device if the user cannot obtain a specific material" means that if the user cannot obtain the necessary material, the system will suggest alternative materials and display that information on the user's screen.

[1538] The system of the present invention is mainly composed of a user terminal, a server, and a database. How each function is implemented will be specifically explained below.

[1539] User Registration and Login

[1540] The user uses the terminal to enter their "user name," "email address," and "password," and sends this information to the server as a registration request. After the server receives the request, it connects to the database and saves the user information. Once the saving is complete, the server returns a response to the terminal indicating successful registration. The user re-enters their "email address" and "password" on the login screen and sends this to the server. The server compares the user information with the information in the database, and if authentication is successful, generates session information and returns it to the terminal.

[1541] Search and browse recipes

[1542] The user enters a keyword (e.g., "sushi") into the search bar of the device and sends a search request. The server receives the request and retrieves the corresponding recipe information from the database. The extracted recipe information is sent to the device, and the user can view a list of recipes as search results.

[1543] Step-by-step guide

[1544] When a user selects a specific recipe, the device sends the selection information to the server. The server then retrieves the step data for that recipe from the database and sends it to the device in sequence. The user can proceed with cooking by checking the displayed steps one by one.

[1545] Proposal of alternative materials

[1546] If a user cannot obtain a particular ingredient, the device sends a request for a substitute ingredient to the server. The server searches the database for the corresponding substitute and sends the obtained alternative to the device. The user can then confirm the suggested substitute and continue cooking.

[1547] Food photo sharing and community support

[1548] After the dish is completed, the user takes a photo of the dish using their device and sends an upload request along with their comments to the server. The server receives the request and stores the photo and comments in a database. The server then updates the community feed and notifies other users' devices, allowing other users to view the photos and comments and post feedback and comments.

[1549] For example, if a user wants to make "sushi," they first log in (see the steps for user registration and login), enter "sushi" in the search bar, and press the search button. The server extracts recipe information related to "sushi" from the database and returns it to the device. The user selects "sushi" from the displayed list of recipes and checks the details. Next, the user proceeds with the cooking while following the step-by-step guide, and if some ingredients are unavailable, they send a request for substitute ingredients to the server and use the suggested substitute ingredients. Once the dish is complete, they upload photos and comments from their device to the server and share them with the community. They can receive feedback and comments from other users to help them with their next cooking.

[1550] (Example of input to a generative AI model)

[1551] "For a Japanese recipe website, please write a paragraph explaining how to suggest alternative ingredients when ingredients cannot be easily sourced."

[1552] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1553] User Registration and Login

[1554] Step 1: Submit a user registration request

[1555] Input: The user enters their username, email address, and password on the device.

[1556] Action: User clicks the "Register" button.

[1557] Output: The device sends a registration request containing the entered information to the server.

[1558] Specific operation: The device converts the input form data into JSON format and sends it to the server as an HTTP POST request.

[1559] Step 2: Save user information

[1560] Input: The server receives a registration request.

[1561] Operation: The server connects to the database and stores the user information.

[1562] Output: If the save is successful, the server sends a registration success response to the device.

[1563] What happens: The server validates the data and executes an INSERT query to the database.

[1564] Step 3: Send a login request

[1565] Input: The user enters their email address and password on the login screen.

[1566] Action: User clicks the "Login" button.

[1567] Output: The device sends a login request containing this information to the server.

[1568] Specific operation: The device converts the input form data into JSON format and sends it to the server as an HTTP POST request.

[1569] Step 4: Authenticate and return session information

[1570] Input: The server receives a login request.

[1571] Operation: The server retrieves the relevant user information from the database and matches it with the input information.

[1572] Output: If authentication is successful, the server generates session information and returns it to the terminal.

[1573] Specific operation: The server executes a SELECT query, and if authentication is successful, generates a session ID, saves it in the database, and then returns the session ID as a response.

[1574] Search and browse recipes

[1575] Step 1: Submit a search request

[1576] Input: A user types a keyword (e.g., "sushi") into the device's search bar.

[1577] Action: User clicks the Search button.

[1578] Output: The terminal sends the entered keyword to the server.

[1579] Specific operation: The device converts the search keywords into JSON format and sends it to the server as an HTTP GET request.

[1580] Step 2: Extract recipe information

[1581] Input: The server receives a search request.

[1582] Operation: The server searches and extracts recipe information that matches the keywords from the database.

[1583] Output: The extracted recipe information is formatted into a recipe list and sent to the terminal.

[1584] Specific operation: The server executes a SELECT query, formats the results in JSON format, and sends them to the terminal as an HTTP response.

[1585] Step 3: Viewing search results

[1586] Input: The device receives a response containing the recipe information.

[1587] Interaction: The device displays a list of recipes to the user.

[1588] Output: User selects recipe they are interested in and checks its details.

[1589] Specific operation: The device parses the received JSON data and displays it in a list format in the user interface.

[1590] Step-by-step guide

[1591] Step 1: Request step information

[1592] Input: The user selects a specific recipe.

[1593] Operation: The device requests information about the selected recipe from the server.

[1594] Output: The device sends a step information request to the server.

[1595] Specific operation: The device sends JSON data including the recipe ID to the server as an HTTP GET request.

[1596] Step 2: Acquire step data

[1597] Input: The server receives a request for step information.

[1598] Operation: The server retrieves the relevant step data (including photos and videos) from the database.

[1599] Output: Sends the acquired step data to the terminal.

[1600] Specific operation: The server retrieves the relevant data using a SELECT query, formats it in JSON format, and returns it to the terminal as a response.

[1601] Step 3: Step-by-step view

[1602] Input: The terminal interprets the received step data.

[1603] Operation: The device displays the cooking steps to the user.

[1604] Output: The user proceeds with cooking by checking the steps displayed on the screen one by one.

[1605] What it does: The device analyzes information step by step and displays text and media content incrementally.

[1606] Proposal of alternative materials

[1607] Step 1: Submit a Substitute Material Request

[1608] Input: User confirms that a specific ingredient is unavailable.

[1609] Operation: The user submits a material substitution request from the terminal.

[1610] Output: The terminal sends the request data for the substitute material to the server.

[1611] Specific operation: The terminal converts the request content into JSON format and sends it to the server as an HTTP GET request.

[1612] Step 2: Get alternatives

[1613] Input: The server receives a request for a substitute material.

[1614] Operation: The server searches and retrieves information on the relevant alternative materials from the database.

[1615] Output: Sends the obtained information about alternative materials to the terminal.

[1616] Specific operation: The server retrieves relevant material information using a SELECT query, formats it in JSON format, and returns it to the terminal.

[1617] Step 3: View alternatives

[1618] Input: The terminal receives a response containing information about alternative ingredients.

[1619] Interaction: The terminal will display the alternative material information to the user.

[1620] Output: User reviews the proposed alternative materials and uses them if necessary.

[1621] Specific behavior: The device parses the received JSON data and displays it in the user interface in an appropriate format.

[1622] Food photo sharing and community support

[1623] Step 1: Upload your photo and comment

[1624] Input: User takes a photo of a dish.

[1625] Operation: A user sends an upload request from their device with a photo and comment.

[1626] Output: The device sends an upload request to the server.

[1627] Specific operation: The device converts the photo and comment into JSON format and sends it to the server as an HTTP POST request.

[1628] Step 2: Save the photo and comment

[1629] Input: The server receives an upload request.

[1630] Operation: The server stores the photos and comments in a database.

[1631] Output: If the save was successful, the server updates the community feed.

[1632] Specific behavior: The server validates the data and executes an INSERT query to save it to the database.

[1633] Step 3: Update your feed and notify

[1634] Input: The server updates the community feed.

[1635] Operation: The server sends notifications to other users' devices.

[1636] Output: Other users will receive notifications of updates to the feed.

[1637] Specific operation: The server converts the update notification into JSON format and sends it to other user devices as an HTTP PUSH notification.

[1638] This concludes the explanation of the specific processing steps of the overall system program.

[1639] (Application example 1)

[1640] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1641] Currently, there are a wide variety of systems for learning Japanese cuisine recipes, but only a limited number of systems provide practical information useful in the kitchen and support the cooking process. In particular, there are no systems that offer integrated functions such as recipe search, suggestions for alternative ingredients, step-by-step cooking guides, and photo sharing of dishes, as well as the ability to search and view restaurant menus. Therefore, there is a need for a system that allows users to efficiently and easily learn and perform cooking in real-world environments such as Japanese restaurants and cooking classes.

[1642] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1643] In this invention, the server includes means for storing recipe data in a database, means for authenticating users, means for searching for recipes, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, means for searching and browsing restaurant menus, and means for checking how to make dishes step by step in Japanese restaurants or cooking classes, thereby enabling users to efficiently and easily learn and practice cooking in real environments such as Japanese restaurants or cooking classes.

[1644] "Recipe data" refers to information including cooking methods, ingredients, and cooking steps.

[1645] A "database" refers to a system for systematically storing and managing information such as recipe data and user information.

[1646] "User authentication" refers to the process of verifying the information entered (email address and password) when a user logs in to a system and confirming that the user is a legitimate user.

[1647] "Recipe search" refers to the function that extracts recipe information from the database based on entered keywords and provides it to the user.

[1648] "Step-by-step display" refers to a function that breaks down the cooking steps of the selected recipe into order, allowing the user to proceed with the cooking process by checking each step.

[1649] "Substitute material" refers to another material that can perform an equivalent function if a particular material is not available.

[1650] "Means for suggesting alternative materials" refers to a function that suggests suitable alternative materials from a database when a user has difficulty obtaining a particular material.

[1651] "Means for uploading and sharing photos of food" refers to the ability for users to upload photos of food they have cooked to the system and share them with other users.

[1652] "Means for displaying feedback and comments" refers to a function that displays ratings and opinions from other users on the screen so that users can check them.

[1653] "Means for searching and viewing restaurant menus" refers to the function of searching for menus offered at restaurants such as Japanese restaurants and displaying their contents.

[1654] "Cooking classes" refers to facilities and services that provide educational environments and programs for learning how to cook.

[1655] This invention is a system that allows users to effectively learn and practice Japanese cuisine at brick-and-mortar establishments such as Japanese restaurants and cooking schools. This system is primarily composed of a server, user terminals, and a user community. Each element is described in detail below.

[1656] 1. User Registration and Login

[1657] Program Generation

[1658] The server has an API endpoint for user registration and login. The user enters their "username," "email address," and "password" from their device and sends them to the server. The server receives this and stores the user information in a database. When the user logs in, they enter their "email address" and "password," which the server authenticates. After successful authentication, session information is generated and returned to the user.

[1659] Natural language explanations

[1660] In order for a user to use the system, they must first register. The user enters their "user name," "email address," and "password" on their terminal and sends a registration request to the server. The server receives this and saves the user information in a database. The "email address" and "password" entered by the user when logging in are compared with the information in the database for authentication, and if successful, session information is returned to the user. This completes user authentication.

[1661] 2. Search and browse recipes

[1662] Program Generation

[1663] When a user searches for a Japanese recipe, they enter a keyword (e.g., "sushi") into the device's search bar and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and check its details.

[1664] Natural language explanations

[1665] When a user searches for Japanese cuisine, they enter specific keywords into the device's search bar and send a request to the server. The server then extracts the relevant recipe information from the database and sends it back to the device. The user can then select the recipe they are interested in from the displayed list and view detailed information.

[1666] 3. Step-by-step guide

[1667] Program Generation

[1668] The system displays the cooking steps for the recipe selected by the user, step by step. Specifically, the server receives the user's request, retrieves the corresponding step data from the database, and sends it to the device in sequence. The data may include photos and videos, and is displayed in a way that makes it easy for the user to understand.

[1669] Natural language explanations

[1670] A guide is provided to help users check the cooking steps for the recipe they have selected. Upon receiving a user's request, the server retrieves the corresponding step data from the database and sends it to the device in sequence. This allows the user to proceed with the cooking process while checking each step of the cooking procedure.

[1671] 4. Proposal of alternative materials

[1672] Program Generation

[1673] If a user has difficulty obtaining a particular ingredient, the user sends a request for a substitute ingredient from the device to the server, which retrieves the corresponding alternative from the database and returns it to the device. The user can then continue cooking using the suggested alternative.

[1674] Natural language explanations

[1675] If a user feels that a particular ingredient is difficult to obtain, they can send a request to the server for alternative ingredient suggestions. The server will then retrieve the relevant alternative ingredient suggestions from the database and send them back to the device, allowing the user to continue cooking using the suggested alternative ingredients.

[1676] 5. Food photo sharing and community support

[1677] Program Generation

[1678] After the dish is complete, the user takes a photo and uploads it to the server from their device along with their comments. The server stores the photo and comments in a database and updates the community feed, which also displays feedback and comments from other users.

[1679] Natural language explanations

[1680] Users take a photo of their finished dish and upload it to the server along with their comments from their device. The server stores the photo and comments in a database and updates the community feed, displaying feedback and comments from other users and encouraging interaction between users.

[1681] Specific examples

[1682] For example, consider a user who wants to make "sushi." After logging in, the user enters "sushi" in the search bar and presses the search button. The server retrieves recipes related to "sushi" from the database and returns the information to the device. The user selects "sushi" from the displayed list and checks the detailed information. Next, the user proceeds with the cooking process while following the step-by-step guide. If some of the ingredients are unavailable, the user can request alternative ingredients from the server and use the suggested substitute ingredients. Once the dish is complete, the user can upload photos and comments from the device to the server and share them with the community. The user can receive feedback and comments from other users and use them to improve their next cooking experience.

[1683] Prompt Sentence Examples

[1684] "A user launches the app and searches for 'sushi.' The app retrieves recipes related to 'sushi' from the database and displays them to the user. When the user selects a particular recipe, the app displays step-by-step cooking instructions. Also, if a particular ingredient is unavailable, the app suggests alternative ingredients."

[1685] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1686] Step 1:

[1687] User Registration and Login

[1688] Input: Username, Email Address, and Password

[1689] Operation: The device sends input data to the server.

[1690] Data processing: The server validates the data it receives and converts it into a format that can be stored in the database.

[1691] Output: The server saves the user information in the database and returns a registration success message to the terminal.

[1692] Step 2:

[1693] User authentication

[1694] Input: User's email address and password

[1695] How it works: The device sends authentication data to the server.

[1696] Data calculation: The server compares the data it receives with the information in the database.

[1697] Output: If authentication is successful, generate session information and return it to the terminal. If authentication fails, return an error message.

[1698] Step 3:

[1699] Search and browse recipes

[1700] Input: User search keyword (e.g. "sushi")

[1701] How it works: The device sends a search request to the server.

[1702] Data processing: The server searches the database based on keywords and extracts relevant recipe information.

[1703] Output: Returns search results to the device and displays a list of recipes to the user.

[1704] Step 4:

[1705] Step-by-step guide

[1706] Input: Recipe ID selected by the user

[1707] How it works: The device sends a request to the server.

[1708] Data calculation: The server retrieves the step data corresponding to the recipe ID from the database.

[1709] Output: Sends step data (including photos and videos) to the device sequentially and displays them to the user.

[1710] Step 5:

[1711] Proposal of alternative materials

[1712] Input: User request for specific material alternatives

[1713] How it works: The device sends a request to the server.

[1714] Data calculation: The server searches and extracts the relevant alternative materials from the database.

[1715] Output: Sends the proposed alternative material information to the terminal.

[1716] Step 6:

[1717] Food photo sharing and community support

[1718] Input: User-taken food photos and comments

[1719] What happens: Your device uploads the photo and comments to the server.

[1720] Data processing: The server receives the photos and comments and stores them in a database.

[1721] Output: Update the community feed and display the new information on other users' devices.

[1722] Examples of concrete examples and prompts

[1723] For example, if a user wants to make "sushi," the process would follow these steps:

[1724] Step 1: The user enters their username, email address, and password on their device and sends them to the server. The server receives them and stores them in a database.

[1725] Step 2: The user enters their email address and password for authentication. The server compares the information with the database and returns session information if authentication is successful.

[1726] Step 3: The user types "sushi" into the search bar and sends a search request to the server, which searches its database for recipes related to "sushi" and returns them to the device.

[1727] Step 4: The user requests the selected recipe from the server and receives step-by-step cooking instructions.

[1728] Step 5: If a particular ingredient is not available, a request for an alternative ingredient is sent to the server, which then suggests an alternative ingredient.

[1729] Step 6: After the dish is completed, the user uploads the photo and comment to the server and shares it with the community. They receive feedback and comments from other users.

[1730] Example prompt sentence:

[1731] "A user launches the app and searches for 'sushi.' The app retrieves recipes related to 'sushi' from the database and displays them to the user. When the user selects a particular recipe, the app displays step-by-step cooking instructions. Also, if a particular ingredient is unavailable, the app suggests alternative ingredients."

[1732] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1733] The system of the present invention includes means for storing recipe data in a database, means for user authentication, means for recipe search, means for displaying step-by-step recipe instructions, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, and an emotion engine for recognizing user emotions.

[1734] User Registration and Login

[1735] In order for a user to use the system, they must first register. The user enters their "user name," "email address," and "password" on their device and sends a registration request to the server. The server receives this and stores the user information in a database. Once registration is complete, the user enters their "email address" and "password" on the login screen and is authenticated. The server compares the information with that in the database and, if successful, returns session information to the user.

[1736] Search and browse recipes

[1737] When a user searches for a Japanese recipe, they enter a keyword (e.g., "sushi") into the device's search bar and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and check its details.

[1738] Step-by-step guide

[1739] The user can check each cooking step on the details screen of the selected recipe. The server receives a request from the device and retrieves the corresponding step data (including photos and videos) from the database and sends them to the device in sequence. The user proceeds with the cooking process by checking each step displayed on the screen.

[1740] Proposal of alternative materials

[1741] If a user has difficulty obtaining a particular ingredient, they need to be offered an alternative. In this case, the device sends a request for an alternative ingredient to the server. The server retrieves the relevant alternative from the database and returns it to the device. The user can then continue cooking using the alternative provided.

[1742] Food photo sharing and community support

[1743] After the dish is complete, the user takes a photo and uploads it to the server along with their comments. The server stores the photo and comments in a database and updates the community feed. This update is then reflected on other users' devices, allowing them to receive feedback and comments from other users. This promotes interaction and support among users.

[1744] Introducing the Emotion Engine

[1745] The app has an emotion engine that recognizes the user's emotions, and can analyze their facial expressions and text comments to suggest suitable recipes and substitute ingredients.

[1746] For example, consider a case where a user wants to make "sushi." After logging in, the user enters "sushi" in the search bar and presses the search button. The server extracts recipes related to "sushi" from the database and returns the information to the device. The user selects "sushi" from the displayed list and checks the detailed information.

[1747] Next, you cook using a step-by-step guide. If you don't have some ingredients, you can request a substitute from the server and use the suggested alternative. Once your dish is complete, you can upload a photo and comments from your device to the server and share them with the community. You can receive feedback and comments from other users to improve your next cooking experience.

[1748] Furthermore, the emotion engine can recognize emotions from the user's facial expressions and comments, suggesting more difficult recipes if the user is having fun, and easier recipes if the user is feeling down. This type of customized feedback is provided by the emotion engine.

[1749] In this way, the system of the present invention supports users in the process of learning and enjoying Japanese cuisine and promotes mutual interaction through the community. Furthermore, the introduction of an emotion engine provides more personalized services and improves user satisfaction.

[1750] The processing flow will be explained below.

[1751] Modes for implementing the invention combining emotion engines

[1752] User Registration and Login

[1753] Step 1:

[1754] User: Enter your "Username", "Email address", and "Password" in the new registration form and press the registration button.

[1755] Step 2:

[1756] Terminal: Converts the input information into JSON format and sends an API request to the server.

[1757] Step 3:

[1758] Server: Receives the API request and validates the input information.

[1759] Step 4:

[1760] Server: After verification, save the new user information to the database.

[1761] Step 5:

[1762] Server: Returns a message to the terminal indicating that saving is complete.

[1763] Step 6:

[1764] Terminal: Display a save complete message to the user.

[1765] Step 7:

[1766] User: Enter your email address and password in the login form and click the login button.

[1767] Step 8:

[1768] Terminal: Converts login information into JSON format and sends an API request to the server.

[1769] Step 9:

[1770] Server: Receives the API request, retrieves the relevant user information from the database, and collates it.

[1771] Step 10:

[1772] Server: If authentication is successful, it creates a user session and returns a session token to the device.

[1773] Step 11:

[1774] Terminal: Receives the session token and logs the user in.

[1775] Search and browse recipes

[1776] Step 1:

[1777] User: Enter a keyword (e.g., "sushi") in the search bar and press the search button.

[1778] Step 2:

[1779] Terminal: Converts search keywords into JSON format and sends an API request to the server.

[1780] Step 3:

[1781] Server: Receives the API request and searches the database for the corresponding recipe information.

[1782] Step 4:

[1783] Server: Converts the relevant recipe information into JSON format and returns it to the device.

[1784] Step 5:

[1785] Terminal: Displays the received recipe information to the user.

[1786] View detailed recipes and step-by-step guides

[1787] Step 1:

[1788] User: Select and click on the recipe you are interested in from the list of recipes displayed.

[1789] Step 2:

[1790] Terminal: Convert the selected recipe ID into JSON format and send an API request to the server.

[1791] Step 3:

[1792] Server: Receives the API request and retrieves the corresponding recipe details from the database.

[1793] Step 4:

[1794] Server: Converts detailed information into JSON format and returns it to the terminal.

[1795] Step 5:

[1796] Terminal: Displays the received details to the user.

[1797] Step 6:

[1798] User: On the recipe details screen, follow the step-by-step guide as you cook.

[1799] Step 7:

[1800] Terminal: Requests next step data from the server based on the user's progress.

[1801] Step 8:

[1802] Server: Retrieves the next step data from the database and returns it to the terminal.

[1803] Step 9:

[1804] Terminal: Display step data to the user.

[1805] Proposal of alternative materials

[1806] Step 1:

[1807] User: Clicks a button to request an alternative material suggestion for a particular material.

[1808] Step 2:

[1809] Terminal: Converts material information into JSON format and sends an API request to the server to request a substitute material.

[1810] Step 3:

[1811] Server: Receives the request and retrieves the relevant information on alternative materials from the database.

[1812] Step 4:

[1813] Server: Converts the alternative material information into JSON format and returns it to the terminal.

[1814] Step 5:

[1815] Terminal: Displays the received alternative material information to the user.

[1816] Food photo sharing and community support

[1817] Step 1:

[1818] User: Take a photo of the dish and enter it along with a comment on the upload screen.

[1819] Step 2:

[1820] Device: Convert the photo and comment into JSON format and send an API request to the server to upload.

[1821] Step 3:

[1822] Server: Receives the request and stores the photos and comments in a database.

[1823] Community Feed Updates

[1824] Step 1:

[1825] Server: Update the community feed every time a new post is made.

[1826] Step 2:

[1827] Device: Receives updated feed information and displays it on the user's screen.

[1828] Step 3:

[1829] Users: View and interact with other users' feedback and comments.

[1830] Introducing the Emotion Engine

[1831] Step 1:

[1832] User: Enter facial expressions or text comments while using the system.

[1833] Step 2:

[1834] Terminal: Sends the user's facial expression data or text comments to the emotion engine.

[1835] Step 3:

[1836] Emotion Engine: Analyzes the received data and identifies the user's emotions.

[1837] Step 4:

[1838] Server: Based on the analysis results from the emotion engine, it suggests suitable recipes and alternative ingredients.

[1839] Step 5:

[1840] On the device: Display suggestions to the user.

[1841] Step 6:

[1842] User: Proceed with cooking using suggested recipes and alternative ingredients.

[1843] For example, if a user wants to make "sushi," they enter "sushi" in the search bar and press the search button. The server retrieves recipes related to "sushi" from the database and returns them to the device. The user selects "sushi" from the displayed list and checks the details. If a substitute ingredient is needed during the cooking process, a request is sent to the server and the suggested ingredient is used. Once the dish is complete, the user can upload a photo and comment and receive feedback from the community.

[1844] Furthermore, the emotion engine recognizes the user's emotions from their facial expressions and comments, and suggests more difficult recipes if they are having fun, and easier recipes if they are feeling down, providing this kind of customized feedback.

[1845] Example 2

[1846] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1847] Previous recipe search systems had issues such as a lack of specific instructions required for users to cook a dish and a lack of suggestions for alternative ingredients. They also lacked functionality for users to share the cooking process and results with other users, and were unable to customize recipes based on the user's feelings. This led to issues such as low user satisfaction and a low rate of continued use of the system.

[1848] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1849] In this invention, the server includes means for storing recipe data in a database, means for authenticating users, means for searching for recipes, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, means for receiving requests from the terminal and retrieving corresponding data from the database, and an emotion engine for identifying and analyzing user emotions. This allows users to efficiently obtain the information necessary for cooking Japanese cuisine and enjoy cooking while interacting with other users. Furthermore, by utilizing the emotion engine, it is possible to suggest optimal recipes and alternative ingredients based on the user's emotions, thereby improving user satisfaction.

[1850] "Recipe data" refers to data that includes information such as cooking steps, necessary ingredients, quantities, cooking methods, etc.

[1851] A "database" is a system or device for efficiently storing, managing, and searching large amounts of data.

[1852] "User authentication" is the process of verifying that a user is authorized to access a system.

[1853] "Recipe search" is the process of extracting desired recipe data from a database based on conditions specified by the user.

[1854] "Step-by-step" refers to showing the recipe cooking steps step by step.

[1855] "Alternative materials" are substitute materials that can be used when a particular material is unavailable or according to user preference.

[1856] "Uploading" is the act of a user sending data from their own device to a server.

[1857] "Sharing" refers to making data uploaded by a user available for viewing and use by other users.

[1858] "Feedback" refers to reaction information such as comments and ratings provided by other users.

[1859] A "request" refers to a user requesting specific information or service from a server.

[1860] An "emotion engine" is a system or algorithm that analyzes emotions from a user's facial expressions and text and makes appropriate suggestions.

[1861] The system of the present invention is a comprehensive system that stores recipe data in a database, authenticates users, and provides multiple functions, such as recipe search, step-by-step recipe instructions, suggests alternative ingredients, and allows users to upload and share photos of their dishes. It also includes a sentiment engine that displays feedback and comments from other users and identifies and analyzes user sentiment.

[1862] System configuration

[1863] Hardware and Software Used

[1864] This system is primarily built using servers, databases, and user devices (smartphones and PCs). The software used includes a web application framework, a database management system, image analysis software, and sentiment analysis algorithms.

[1865] Servers: Includes web servers, database servers, and AI engine servers.

[1866] Database: Use SQL database or NoSQL database.

[1867] User devices: Smartphones, PCs, tablets, etc.

[1868] Software: Web application frameworks (e.g., Django, Ruby on Rails), database management systems (e.g., MySQL, PostgreSQL), image analysis software (e.g., OpenCV), sentiment analysis algorithms (e.g., IBM Watson, Google Cloud Natural Language API)

[1869] Explanation of program processing

[1870] User Registration and Login

[1871] When a user uses the system, they must first register. The user enters their "user name," "email address," and "password" from their device and sends a registration request to the server. The server receives this and stores the user information in a database. Once registration is complete, the user enters their "email address" and "password" on the login screen and is authenticated. The server compares the information with that in the database and, if successful, returns session information to the user.

[1872] Examples:

[1873] Prompt statement:

[1874] Enter your email address:

[1875] Please enter your password:

[1876] Enter your username:

[1877] Search and browse recipes

[1878] When a user searches for a recipe, they enter a keyword (e.g., "sushi") into the search bar on their device and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and view its details.

[1879] Examples:

[1880] Prompt statement:

[1881] Enter the name of the dish you want to search for recipes for:

[1882] Example: Sushi

[1883] Step-by-step guide

[1884] The user can check each cooking step on the details screen of the selected recipe. The server receives a request from the device and retrieves the corresponding step data (including photos and videos) from the database and sends them to the device in sequence. The user proceeds with the cooking process by checking each step displayed on the screen.

[1885] Proposal of alternative materials

[1886] If a user has difficulty obtaining a particular ingredient, they need to be offered an alternative. In this case, the device sends a request for an alternative ingredient to the server. The server retrieves the relevant alternative from the database and returns it to the device. The user can then continue cooking using the alternative provided.

[1887] Food photo sharing and community support

[1888] After the dish is complete, the user takes a photo and uploads it to the server along with their comments. The server stores the photo and comments in a database and updates the community feed. This update is then reflected on other users' devices, allowing them to receive feedback and comments from other users. This promotes interaction and support among users.

[1889] Introducing the Emotion Engine

[1890] The system has an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's emotions from their facial expressions and text comments, and can suggest appropriate recipes and alternative ingredients. Specifically, if a user comments, "I'm in a good mood today!", the system will suggest more difficult recipes. Conversely, if a user comments, "I'm tired today," the system will suggest easier recipes.

[1891] Specific examples

[1892] Here is a specific use case where the user wants to make sushi:

[1893] 1. User registration and login: Enter your username, email address, and password to register in the system, then log in to use the system.

[1894] 2. Recipe search: Enter "sushi" into the search bar on the device. The server retrieves recipes related to "sushi" from the database and returns the information to the device.

[1895] 3. Step-by-step guide: Users choose a sushi recipe and follow the step-by-step cooking instructions as they go through the process.

[1896] 4. Suggestion of alternative materials: If some of the required materials are not on hand, a request for alternative materials is made to the system, and the server proposes alternatives.

[1897] 5. Share your cooking photos: Once you've finished cooking, upload photos and comments to share with other users and get feedback.

[1898] 6. Use of emotion engine: Recognize emotions from the user's facial expressions and comments and suggest appropriate recipes.

[1899] In this way, the system of the present invention comprehensively supports users in the process of learning and enjoying cooking, and also promotes interaction with other users. Furthermore, the introduction of an emotion engine makes it possible to provide personalized services according to the user's emotions, thereby improving user satisfaction.

[1900] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1901] User Registration and Login

[1902] Step 1:

[1903] Input: The user enters their "username," "email address," and "password" on the device.

[1904] Processing: The terminal sends a registration request containing this information to the server.

[1905] Output: None.

[1906] Specific operation: When the user clicks the "New Registration" button, the terminal creates and sends a request to send the entered information to the server.

[1907] Step 2:

[1908] Input: The server receives a registration request from the terminal.

[1909] Processing: The server stores the received user information in a database.

[1910] Output: Registration successful message.

[1911] Specific operation: After the server saves the user information in the database, it returns a registration success message to the terminal.

[1912] Step 3:

[1913] Input: The user enters their email address and password on the login screen.

[1914] Processing: The terminal sends a login request containing this information to the server.

[1915] Output: None.

[1916] Specific operation: When you click the "Login" button on the login screen, the device creates a login request based on the information entered and sends it to the server.

[1917] Step 4:

[1918] Input: The server receives a login request.

[1919] Processing: The server checks the user information against the database and performs authentication.

[1920] Output: If authentication is successful, session information is generated and returned to the terminal.

[1921] What happens: The server checks the user's email address and password against records in its database, and if authentication is successful, generates and returns session information.

[1922] Search and browse recipes

[1923] Step 1:

[1924] Input: The user enters a keyword (e.g., "sushi") into the device's search bar.

[1925] Processing: The terminal sends a search request to the server.

[1926] Output: None.

[1927] Specific operation: When the user clicks the "Search" button, the device creates a search request based on the entered keywords and sends it to the server.

[1928] Step 2:

[1929] Input: The server receives a search request.

[1930] Processing: The server extracts the relevant recipe information from the database.

[1931] Output: Relevant recipe information.

[1932] Specific operation: The server searches the recipe database and extracts recipe information that matches the keywords in list form.

[1933] Step 3:

[1934] Input: The server sends the extracted recipe information to the terminal.

[1935] Processing: The device displays the received recipe information to the user.

[1936] Output: A list of displayed recipes.

[1937] Specific operation: The terminal displays the recipe information received from the server on the screen.

[1938] Step 4:

[1939] Input: The user selects the recipe of interest from the displayed list of recipes.

[1940] Processing: The device sends a request to the server to display the details of the selected recipe.

[1941] Output: None.

[1942] Specific operation: When the user clicks on the selected recipe, the device requests detailed information about that recipe from the server.

[1943] Step-by-step guide

[1944] Step 1:

[1945] Input: User requests cooking instructions on the recipe details screen.

[1946] Processing: The terminal sends a cooking instructions request to the server.

[1947] Output: None.

[1948] Specific operation: When the user clicks the "Start Cooking" button, the terminal creates a cooking procedure request and sends it to the server.

[1949] Step 2:

[1950] Input: The server receives a cooking instructions request.

[1951] Processing: The server retrieves cooking recipe data from the database.

[1952] Output: Cooking instructions data.

[1953] Specific operation: The server retrieves the cooking procedure data for the corresponding recipe from the database and returns it to the terminal along with photo and video information.

[1954] Step 3:

[1955] Input: The server sends the acquired cooking procedure data to the terminal.

[1956] Processing: The terminal displays the received procedure data to the user in sequence.

[1957] Output: Displayed cooking instructions.

[1958] Specific operation: The device receives cooking instructions from the server and displays them on the screen step by step. The user proceeds with the cooking process by checking each step one by one.

[1959] Proposal of alternative materials

[1960] Step 1:

[1961] Input: If a user does not have a specific ingredient available, they can submit a request for a substitute ingredient from their device.

[1962] Processing: The terminal sends a substitute material request to the server.

[1963] Output: None.

[1964] Specific operation: When the user clicks the "Search for alternative materials" button, the terminal creates a request for alternative materials and sends it to the server.

[1965] Step 2:

[1966] Input: The server receives a substitute material request.

[1967] Processing: The server retrieves the relevant alternative material information from the database.

[1968] Output: Alternative material information.

[1969] Specific operation: The server retrieves alternatives to the requested ingredients from the database (e.g., tuna as an alternative to salmon) and returns them to the terminal.

[1970] Step 3:

[1971] Input: The server sends the acquired alternative material information to the terminal.

[1972] Processing: The terminal displays the received alternative material information to the user.

[1973] Output: Displayed alternative material information.

[1974] Specific operation: The device displays the information on alternative ingredients received from the server on the screen, and the user can use it as a reference to continue cooking.

[1975] Food photo sharing and community support

[1976] Step 1:

[1977] Input: The user takes a photo of the finished dish and enters a comment.

[1978] Processing: The device sends an upload request containing the photo and comment to the server.

[1979] Output: None.

[1980] Specific operation: When the user clicks the "Upload Photo" button, the device will send the taken photo and any comments entered to the server.

[1981] Step 2:

[1982] Input: The server receives an upload request.

[1983] Processing: The server stores the photos and comments in a database.

[1984] Output: Save complete message.

[1985] Specific operation: The server saves the received photos and comments in a database and returns a save completion message to the terminal.

[1986] Step 3:

[1987] Input: The server updates the community feed based on the stored photos and comments.

[1988] Processing: The server creates new feed data and sends it to other user devices.

[1989] Output: The updated community feed.

[1990] Specific operation: The server updates the community feed based on the user's upload and notifies other users of the data.

[1991] Introducing the Emotion Engine

[1992] Step 1:

[1993] Input: User facial expressions and text comments.

[1994] Processing: The server collects emotion data from users' facial expressions and text comments.

[1995] Output: Emotion data.

[1996] How it works: When a user types a comment or uploads a photo, the system sends a request to the emotion engine to analyze the user's facial expressions and comments.

[1997] Step 2:

[1998] Input: Emotion data.

[1999] Processing: The emotion engine analyzes the collected data and identifies the user's emotional state.

[2000] Output: Sentiment analysis results.

[2001] Specific behavior: The emotion engine analyzes the collected data and identifies emotional states, such as "happy" or "tired."

[2002] Step 3:

[2003] Input: Sentiment analysis results.

[2004] Processing: The server suggests appropriate recipes and alternative ingredients based on the sentiment analysis results.

[2005] Output: Sentiment-based suggestion information.

[2006] Specific operation: The server receives the sentiment analysis results and suggests difficult or easy recipes to the user based on them.

[2007] In this way, the system of the present invention comprehensively supports users in the process of learning and enjoying cooking, and also promotes interaction with other users. Furthermore, the introduction of an emotion engine makes it possible to provide personalized services according to the user's emotions, thereby improving user satisfaction.

[2008] (Application example 2)

[2009] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2010] Conventional online recipe systems provide basic functionality for users to search for recipes and follow the instructions to cook, but this alone is insufficient. With the recent proliferation of food delivery services, there is a demand for systems that comprehensively support the process from ordering food to receiving it and enjoying it. Furthermore, there is a need for systems that can appropriately suggest alternative ingredients when ingredients are out of stock, and make personalized suggestions based on the user's emotions. Conventional systems lack such advanced functionality, making improving the user experience a challenge.

[2011] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2012] In this invention, the server includes means for storing recipe data in a database, means for user authentication, means for searching for recipes, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, means for providing step-by-step guidance from receiving to enjoying the ordered dish, means for suggesting alternative ingredients if an ingredient is out of stock at the time of ordering, and means for recognizing user emotions and providing personalized service. This allows users to enjoy the cooking process with guidance even after ordering and receiving the dish. Furthermore, by providing appropriate alternative suggestions even if an ingredient is out of stock, and by making personalized suggestions based on the user's emotions, a significant improvement in the user experience can be expected.

[2013] "Recipe data" is a collection of information stored in a database, including cooking instructions, necessary ingredients, and steps.

[2014] "User authentication" is the process of verifying the identification information (user name, password, etc.) of a system user and verifying that the user is a legitimate user.

[2015] "Recipe Search" is a system feature that allows users to search for information about specific dishes or ingredients.

[2016] "Step-by-step display" is a system function that guides users through each step of a recipe, checking them one by one.

[2017] "Alternative material suggestions" is a system function that suggests alternative materials when a user has difficulty obtaining a particular material.

[2018] "Means to upload and share photos of food" refers to a function that allows users to take photos of the food they have made and share them with other users on the system.

[2019] "Show feedback and comments" is a system feature that displays opinions and comments from other users.

[2020] "Step-by-step guidance" is a system function that provides detailed explanations and guidance for users to enjoy the food after the ordered food arrives.

[2021] An "emotion recognition engine" is a system function that analyzes emotions from a user's facial expressions and text comments and provides personalized services based on that.

[2022] "Personalized service provision" refers to the system's ability to provide users with information and suggestions that are best suited to them based on their specific interests and emotions.

[2023] The following system configuration is used as an embodiment of the present invention.

[2024] System Overview

[2025] The system includes a database, user authentication mechanism, recipe search function, step-by-step guidance function, ingredient substitution suggestion function, community function, emotion recognition engine, and personalized service provision function, providing comprehensive support that allows users to enjoy the cooking process from ordering to receiving their food.

[2026] Hardware and software used

[2027] Hardware: Smartphones, smart glasses, head-mounted displays, servers

[2028] Software: Flask (a Python web framework), SQLAlchemy (for database operations), emotion recognition engine (facial expression analysis and text analysis)

[2029] Details of each function

[2030] Building a database

[2031] The server stores recipe data in a database, which includes the name of the dish, the ingredients needed, the cooking instructions, and photos or videos. The database is operated using SQLAlchemy.

[2032] User authentication

[2033] In order for a user to use the system, they must register and log in. The authentication information (username, email address, password) is verified by the server, and session information is generated after authentication.

[2034] Recipe Search

[2035] The user enters keywords into the search bar and sends a recipe search request to the server. The server searches the database for the relevant recipe and returns it to the device. For example, it is possible to search for the keyword "sushi."

[2036] Step-by-step guide

[2037] The recipe details screen for a user's selected recipe displays step-by-step cooking instructions, each supplemented with photos and videos to guide the user through the cooking process.

[2038] Proposal of alternative materials

[2039] If some of the ingredients are unavailable, the user can request an alternative, and the server will retrieve the appropriate alternative from the database and return it to the user's device.

[2040] Community Features

[2041] Once the dish is complete, the user can take a photo and upload it along with a comment, which the server stores in a database and displays to other users.

[2042] Emotion Recognition Engine

[2043] Emotions are analyzed from the user's facial expressions and text comments, allowing the server to adjust the difficulty of recipes and make personalized suggestions based on the user's emotional state.

[2044] Specific examples

[2045] For example, if a user wants to make "sushi," they follow these steps: They log in to the application and type "sushi" into the search bar to search for related recipes. They select a sushi recipe that appears and follow a step-by-step guide to cook it. If some of the ingredients are unavailable, they request alternative ingredients and accept suggestions from the server. Once cooking is complete, they take a photo of the dish and share it with the community along with a comment. Furthermore, an emotion recognition engine analyzes the user's facial expressions and comments to see if they are enjoying the dish, and can suggest a recipe with an adjusted difficulty for next time.

[2046] Prompt Sentence Examples

[2047] "Just search for the name of the recipe you're interested in and get detailed step-by-step instructions on how to make it. For example, 'Sushi' or 'Curry'."

[2048] In this way, the system of the present invention can assist users in the process of enjoying cooking and improve the user experience.

[2049] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2050] Step 1:

[2051] A user accesses the system and registers.

[2052] Input: The user enters their "username," "email address," and "password" from their device and sends a registration request to the server.

[2053] Processing: The server stores the received information in a database.

[2054] Output: The server returns a registration completion message to the user terminal.

[2055] Specific operation: A user opens the application and accesses the new registration screen. They enter the required information and press the "Register" button. The server receives this and saves the user information to the database using an SQL query.

[2056] Step 2:

[2057] A user logs in with an existing account.

[2058] Input: The user enters their email address and password on the device and sends a login request to the server.

[2059] Processing: The server checks the information in its database and performs authentication.

[2060] Output: If the authentication is successful, the server generates session information and returns it to the user terminal.

[2061] Specific operation: The user goes to the login screen, enters the required authentication information, and presses the "Login" button. The server receives this and compares it with the information in the database. If the comparison is successful, the server generates session information and returns it to the user's device.

[2062] Step 3:

[2063] A user searches for a recipe.

[2064] Input: The user enters a keyword (e.g., "sushi") into the search bar of the device and sends a search request to the server.

[2065] Processing: The server extracts the relevant recipe information from the database.

[2066] Output: The server returns a list of search results to the user's device.

[2067] How it works: The user enters keywords into the search bar and presses the "Search" button. The server executes an SQL query to retrieve related recipe information from the database and returns the results to the user's device.

[2068] Step 4:

[2069] The user reviews the recipe details and follows the step-by-step instructions.

[2070] Input: Request details of the recipe selected by the user.

[2071] Processing: The server retrieves data for each step (including photos and videos) from the database.

[2072] Output: The server sends step-by-step instructions to the user terminal sequentially.

[2073] Specific operation: The user selects a recipe of interest from the displayed list of recipes and presses the details button. The server retrieves the corresponding step data from the database and sequentially sends it to the user's terminal.

[2074] Step 5:

[2075] If materials are unavailable, suggest alternative materials.

[2076] Input: The user submits a request for a substitute material from the terminal.

[2077] Processing: The server retrieves data on the corresponding alternative materials from the database.

[2078] Output: The server returns the alternative material suggestions to the user terminal.

[2079] Specific operation: If a user does not have a specific ingredient, they request an alternative to that ingredient. The server executes an SQL query to retrieve the alternative ingredient information from the database and returns it to the user's device.

[2080] Step 6:

[2081] Users upload photos of their dishes and share them with the community.

[2082] Input: After the user has finished cooking, they take a photo and send an upload request to the server along with their comments.

[2083] Processing: The server stores the received photos and comments in a database and updates the community feed.

[2084] Output: The server reflects the updated feed information to the user's terminal.

[2085] What it does: Users use the application's "Photo Upload" feature to submit photos of their finished dishes along with their comments. The server receives these, stores them in a database, and updates the community feed, which also displays feedback and comments from other users.

[2086] Step 7:

[2087] Recognize user emotions using an emotion engine.

[2088] Input: The user's facial expression data and text comments are taken as input.

[2089] Processing: The emotion engine analyzes the input data and determines the user's emotional state.

[2090] Output: Display recipe difficulty adjustment and personalized serving suggestions based on emotions.

[2091] Specific operation: The user's facial expression is captured by a camera or a comment is entered. The emotion engine analyzes this data to determine their emotional state (happy, sad, excited, etc.) and suggests the most appropriate service or recipe based on the results. For example, if the user looks happy, it will suggest a more difficult recipe, and conversely, if the user looks depressed, it will suggest an easy recipe.

[2092] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2093] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2094] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2095] [Fourth embodiment]

[2096] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2097] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2098] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2099] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2100] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2101] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2102] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2103] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2104] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2105] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2106] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2107] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2108] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2109] The system of the present invention is mainly composed of a server, user terminals, and a user community. Each element will be explained separately below.

[2110] User Registration and Login

[2111] In order for a user to use the system, they must first register. At this time, the user enters their "user name," "email address," and "password" from their device and sends a registration request to the server. The server receives this and stores the user information in a database. Once registration is complete, the user enters their "email address" and "password" on the login screen and is authenticated. The server compares the information with that in the database and, if successful, returns session information to the user.

[2112] Search and browse recipes

[2113] When a user searches for a Japanese recipe, they enter a keyword (e.g., "sushi") into the device's search bar and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and check its details.

[2114] Step-by-step guide

[2115] For the recipe selected by the user, each cooking step can be checked on the details screen. Upon receiving a request from the device, the server retrieves the corresponding step data (including photos and videos) from the database and sends them to the device in sequence. The user proceeds with the cooking process by checking each step displayed on the screen.

[2116] Proposal of alternative materials

[2117] If a user has difficulty obtaining a particular ingredient, they need to be offered an alternative. In this case, the device sends a request for an alternative ingredient to the server. The server retrieves the relevant alternative from the database and returns it to the device. The user can then continue cooking using the alternative provided.

[2118] Food photo sharing and community support

[2119] After the dish is complete, the user takes a photo and uploads it to the server along with their comments. The server stores the photo and comments in a database and updates the community feed. This update is then reflected on other users' devices, allowing them to receive feedback and comments from other users. This promotes interaction and support among users.

[2120] Specific examples

[2121] For example, consider a case where a user wants to make "sushi." After logging in, the user enters "sushi" in the search bar and presses the search button. The server extracts recipes related to "sushi" from the database and returns the information to the device. The user selects "sushi" from the displayed list and checks the detailed information.

[2122] Next, you cook using a step-by-step guide. If you don't have some ingredients, you can request a substitute from the server and use the suggested alternative. Once your dish is complete, you can upload a photo and comments from your device to the server and share them with the community. You can receive feedback and comments from other users to improve your next cooking experience.

[2123] In this way, the system of the present invention supports users in the process of learning and enjoying Japanese cuisine, and promotes mutual interaction through the community.

[2124] The processing flow will be explained below.

[2125] User Registration and Login

[2126] User Registration

[2127] Step 1:

[2128] User: Enter your "Username", "Email address", and "Password" in the new registration form and press the registration button.

[2129] Step 2:

[2130] Device: Converts registration information into JSON format and sends an API request to the server.

[2131] Step 3:

[2132] Server: Receives the API request and checks the format and content of the input data.

[2133] Step 4:

[2134] Server: Save the new user information to the database.

[2135] Step 5:

[2136] Server: If the save is successful, it returns a success message to the terminal.

[2137] Step 6:

[2138] Terminal: Receives the success message and displays it to the user.

[2139] Log in

[2140] Step 1:

[2141] User: Enter your email address and password in the login form and click the login button.

[2142] Step 2:

[2143] Terminal: Converts login information into JSON format and sends an API request to the server.

[2144] Step 3:

[2145] Server: Receives the API request and retrieves the corresponding user information from the database.

[2146] Step 4:

[2147] Server: Compares the acquired user information with the entered information, and performs authentication if they match.

[2148] Step 5:

[2149] Server: If authentication is successful, it generates session information and returns it to the terminal.

[2150] Step 6:

[2151] Terminal: Receives session information and logs the user in.

[2152] Search and browse recipes

[2153] Recipe Search

[2154] Step 1:

[2155] User: Enter a keyword (e.g., "sushi") in the search bar and press the search button.

[2156] Step 2:

[2157] Terminal: Converts search keywords into JSON format and sends an API request to the server.

[2158] Step 3:

[2159] Server: Receives the API request and searches the database for the corresponding recipe information.

[2160] Step 4:

[2161] Server: Converts the relevant recipe information into JSON format and returns it to the device.

[2162] Step 5:

[2163] Terminal: Displays the received recipe information to the user.

[2164] View recipe details

[2165] Step 1:

[2166] User: Select and click on the recipe you are interested in from the list of recipes displayed.

[2167] Step 2:

[2168] Terminal: Convert the selected recipe ID into JSON format and send an API request to the server.

[2169] Step 3:

[2170] Server: Receives the API request and retrieves the corresponding recipe details from the database.

[2171] Step 4:

[2172] Server: Converts detailed information into JSON format and returns it to the terminal.

[2173] Step 5:

[2174] Terminal: Displays the received details to the user.

[2175] Step-by-step guide

[2176] View Guide

[2177] Step 1:

[2178] User: On a recipe detail screen, clicks a link or button that displays a step-by-step guide.

[2179] Step 2:

[2180] Terminal: Convert the recipe ID into JSON format and send an API request for step data to the server.

[2181] Step 3:

[2182] Server: Receives the API request and retrieves the corresponding step data from the database.

[2183] Step 4:

[2184] Server: Converts step data (including photos and videos) into JSON format and returns it to the device.

[2185] Step 5:

[2186] Terminal: Displays the received step data to the user.

[2187] Proposal of alternative materials

[2188] Request an alternative material

[2189] Step 1:

[2190] User: Clicks a button to request an alternative material suggestion for a particular material.

[2191] Step 2:

[2192] Terminal: Converts material information into JSON format and sends an API request to the server to request a substitute material.

[2193] Step 3:

[2194] Server: Receives the request and retrieves the relevant information on alternative materials from the database.

[2195] Step 4:

[2196] Server: Converts the alternative material information into JSON format and returns it to the terminal.

[2197] Step 5:

[2198] Terminal: Displays the received alternative material information to the user.

[2199] Food photo sharing and community support

[2200] Upload a photo

[2201] Step 1:

[2202] User: Take a photo of the dish and enter it along with a comment on the upload screen.

[2203] Step 2:

[2204] Device: Convert the photo and comment into JSON format and send an API request to the server to upload.

[2205] Step 3:

[2206] Server: Receives the request and stores the photos and comments in a database.

[2207] Community Feed Updates

[2208] Step 1:

[2209] Server: Update the community feed every time a new post is made.

[2210] Step 2:

[2211] Device: Receives updated feed information and displays it on the user's screen.

[2212] Step 3:

[2213] Users: View and interact with other users' feedback and comments.

[2214] The above are the specific processing steps for carrying out the present invention.

[2215] Example 1

[2216] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2217] With conventional recipe search systems, it was difficult for users to find the recipe they were looking for, and they lacked a function to suggest alternative ingredients when ingredients were unavailable, which often hindered the progress of cooking.In addition, interaction between users was limited, and they were unable to receive feedback or comments from other users, which made it difficult to improve their cooking skills or discover new recipes.

[2218] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2219] In this invention, the server includes means for storing user information in a database when a user registers, means for authenticating the user, means for searching and extracting recipe data from the database, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for the user to upload and share photos of the dish, means for displaying feedback and comments from other users, and means for displaying alternative ingredients on the user's terminal if the user cannot obtain a particular ingredient. This allows users to easily search and view desired recipes and receive suggestions for alternative ingredients even if the ingredients are unavailable. Furthermore, sharing photos of dishes promotes interaction within the community, and by receiving feedback and comments from other users, users can improve their cooking skills.

[2220] "User registration" refers to the process by which a user provides their own information to the system and is authenticated as a user.

[2221] "Storing in a database" refers to the persistent storage of digital information so that it can be accessed at a later time.

[2222] "User authentication" is the process by which a system verifies a user's identity and grants appropriate access privileges.

[2223] "Searching and extracting recipe data from a database" means searching for recipe information in a database based on specific conditions and extracting the corresponding data.

[2224] "Step-by-step" means showing the user steps or operations in a sequential, step-by-step manner.

[2225] "Suggesting alternative materials" means suggesting alternative materials that provide the same effect if the user cannot obtain a particular material.

[2226] "Users upload and share photos of their dishes" means that users send photos of their own dishes to the system and share the information with other users.

[2227] "Displaying feedback and comments from other users" means that the system collects opinions and reactions from other users and presents them in a visible form to the user.

[2228] "Displaying alternative materials on the user's device if the user cannot obtain a specific material" means that if the user cannot obtain the necessary material, the system will suggest alternative materials and display that information on the user's screen.

[2229] The system of the present invention is mainly composed of a user terminal, a server, and a database. How each function is implemented will be specifically explained below.

[2230] User Registration and Login

[2231] The user uses the terminal to enter their "user name," "email address," and "password," and sends this information to the server as a registration request. After the server receives the request, it connects to the database and saves the user information. Once the saving is complete, the server returns a response to the terminal indicating successful registration. The user re-enters their "email address" and "password" on the login screen and sends this to the server. The server compares the user information with the information in the database, and if authentication is successful, generates session information and returns it to the terminal.

[2232] Search and browse recipes

[2233] The user enters a keyword (e.g., "sushi") into the search bar of the device and sends a search request. The server receives the request and retrieves the corresponding recipe information from the database. The extracted recipe information is sent to the device, and the user can view a list of recipes as search results.

[2234] Step-by-step guide

[2235] When a user selects a specific recipe, the device sends the selection information to the server. The server then retrieves the step data for that recipe from the database and sends it to the device in sequence. The user can proceed with cooking by checking the displayed steps one by one.

[2236] Proposal of alternative materials

[2237] If a user cannot obtain a particular ingredient, the device sends a request for a substitute ingredient to the server. The server searches the database for the corresponding substitute and sends the obtained alternative to the device. The user can then confirm the suggested substitute and continue cooking.

[2238] Food photo sharing and community support

[2239] After the dish is completed, the user takes a photo of the dish using their device and sends an upload request along with their comments to the server. The server receives the request and stores the photo and comments in a database. The server then updates the community feed and notifies other users' devices, allowing other users to view the photos and comments and post feedback and comments.

[2240] For example, if a user wants to make "sushi," they first log in (see the steps for user registration and login), enter "sushi" in the search bar, and press the search button. The server extracts recipe information related to "sushi" from the database and returns it to the device. The user selects "sushi" from the displayed list of recipes and checks the details. Next, the user proceeds with the cooking while following the step-by-step guide, and if some ingredients are unavailable, they send a request for substitute ingredients to the server and use the suggested substitute ingredients. Once the dish is complete, they upload photos and comments from their device to the server and share them with the community. They can receive feedback and comments from other users to help them with their next cooking.

[2241] (Example of input to a generative AI model)

[2242] "For a Japanese recipe website, please write a paragraph explaining how to suggest alternative ingredients when ingredients cannot be easily sourced."

[2243] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2244] User Registration and Login

[2245] Step 1: Submit a user registration request

[2246] Input: The user enters their username, email address, and password on the device.

[2247] Action: User clicks the "Register" button.

[2248] Output: The device sends a registration request containing the entered information to the server.

[2249] Specific operation: The device converts the input form data into JSON format and sends it to the server as an HTTP POST request.

[2250] Step 2: Save user information

[2251] Input: The server receives a registration request.

[2252] Operation: The server connects to the database and stores the user information.

[2253] Output: If the save is successful, the server sends a registration success response to the device.

[2254] What happens: The server validates the data and executes an INSERT query to the database.

[2255] Step 3: Send a login request

[2256] Input: The user enters their email address and password on the login screen.

[2257] Action: User clicks the "Login" button.

[2258] Output: The device sends a login request containing this information to the server.

[2259] Specific operation: The device converts the input form data into JSON format and sends it to the server as an HTTP POST request.

[2260] Step 4: Authenticate and return session information

[2261] Input: The server receives a login request.

[2262] Operation: The server retrieves the relevant user information from the database and matches it with the input information.

[2263] Output: If authentication is successful, the server generates session information and returns it to the terminal.

[2264] Specific operation: The server executes a SELECT query, and if authentication is successful, generates a session ID, saves it in the database, and then returns the session ID as a response.

[2265] Search and browse recipes

[2266] Step 1: Submit a search request

[2267] Input: A user types a keyword (e.g., "sushi") into the device's search bar.

[2268] Action: User clicks the Search button.

[2269] Output: The terminal sends the entered keyword to the server.

[2270] Specific operation: The device converts the search keywords into JSON format and sends it to the server as an HTTP GET request.

[2271] Step 2: Extract recipe information

[2272] Input: The server receives a search request.

[2273] Operation: The server searches and extracts recipe information that matches the keywords from the database.

[2274] Output: The extracted recipe information is formatted into a recipe list and sent to the terminal.

[2275] Specific operation: The server executes a SELECT query, formats the results in JSON format, and sends them to the terminal as an HTTP response.

[2276] Step 3: Viewing search results

[2277] Input: The device receives a response containing the recipe information.

[2278] Interaction: The device displays a list of recipes to the user.

[2279] Output: User selects recipe they are interested in and checks its details.

[2280] Specific operation: The device parses the received JSON data and displays it in a list format in the user interface.

[2281] Step-by-step guide

[2282] Step 1: Request step information

[2283] Input: The user selects a specific recipe.

[2284] Operation: The device requests information about the selected recipe from the server.

[2285] Output: The device sends a step information request to the server.

[2286] Specific operation: The device sends JSON data including the recipe ID to the server as an HTTP GET request.

[2287] Step 2: Acquire step data

[2288] Input: The server receives a request for step information.

[2289] Operation: The server retrieves the relevant step data (including photos and videos) from the database.

[2290] Output: Sends the acquired step data to the terminal.

[2291] Specific operation: The server retrieves the relevant data using a SELECT query, formats it in JSON format, and returns it to the terminal as a response.

[2292] Step 3: Step-by-step view

[2293] Input: The terminal interprets the received step data.

[2294] Operation: The device displays the cooking steps to the user.

[2295] Output: The user proceeds with cooking by checking the steps displayed on the screen one by one.

[2296] What it does: The device analyzes information step by step and displays text and media content incrementally.

[2297] Proposal of alternative materials

[2298] Step 1: Submit a Substitute Material Request

[2299] Input: User confirms that a specific ingredient is unavailable.

[2300] Operation: The user submits a material substitution request from the terminal.

[2301] Output: The terminal sends the request data for the substitute material to the server.

[2302] Specific operation: The terminal converts the request content into JSON format and sends it to the server as an HTTP GET request.

[2303] Step 2: Get alternatives

[2304] Input: The server receives a request for a substitute material.

[2305] Operation: The server searches and retrieves information on the relevant alternative materials from the database.

[2306] Output: Sends the obtained information about alternative materials to the terminal.

[2307] Specific operation: The server retrieves relevant material information using a SELECT query, formats it in JSON format, and returns it to the terminal.

[2308] Step 3: View alternatives

[2309] Input: The terminal receives a response containing information about alternative ingredients.

[2310] Interaction: The terminal will display the alternative material information to the user.

[2311] Output: User reviews the proposed alternative materials and uses them if necessary.

[2312] Specific behavior: The device parses the received JSON data and displays it in the user interface in an appropriate format.

[2313] Food photo sharing and community support

[2314] Step 1: Upload your photo and comment

[2315] Input: User takes a photo of a dish.

[2316] Operation: A user sends an upload request from their device with a photo and comment.

[2317] Output: The device sends an upload request to the server.

[2318] Specific operation: The device converts the photo and comment into JSON format and sends it to the server as an HTTP POST request.

[2319] Step 2: Save the photo and comment

[2320] Input: The server receives an upload request.

[2321] Operation: The server stores the photos and comments in a database.

[2322] Output: If the save was successful, the server updates the community feed.

[2323] Specific behavior: The server validates the data and executes an INSERT query to save it to the database.

[2324] Step 3: Update your feed and notify

[2325] Input: The server updates the community feed.

[2326] Operation: The server sends notifications to other users' devices.

[2327] Output: Other users will receive notifications of updates to the feed.

[2328] Specific operation: The server converts the update notification into JSON format and sends it to other user devices as an HTTP PUSH notification.

[2329] This concludes the explanation of the specific processing steps of the overall system program.

[2330] (Application example 1)

[2331] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2332] Currently, there are a wide variety of systems for learning Japanese cuisine recipes, but only a limited number of systems provide practical information useful in the kitchen and support the cooking process. In particular, there are no systems that offer integrated functions such as recipe search, suggestions for alternative ingredients, step-by-step cooking guides, and photo sharing of dishes, as well as the ability to search and view restaurant menus. Therefore, there is a need for a system that allows users to efficiently and easily learn and perform cooking in real-world environments such as Japanese restaurants and cooking classes.

[2333] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2334] In this invention, the server includes means for storing recipe data in a database, means for authenticating users, means for searching for recipes, means for displaying step-by-step recipe procedures, means for suggesting alternative ingredients, means for users to upload and share photos of dishes, means for displaying feedback and comments from other users, means for searching and browsing restaurant menus, and means for checking how to make dishes step by step in Japanese restaurants or cooking classes, thereby enabling users to efficiently and easily learn and practice cooking in real environments such as Japanese restaurants or cooking classes.

[2335] "Recipe data" refers to information including cooking methods, ingredients, and cooking steps.

[2336] A "database" refers to a system for systematically storing and managing information such as recipe data and user information.

[2337] "User authentication" refers to the process of verifying the information entered (email address and password) when a user logs in to a system and confirming that the user is a legitimate user.

[2338] "Recipe search" refers to the function that extracts recipe information from the database based on entered keywords and provides it to the user.

[2339] "Step-by-step display" refers to a function that breaks down the cooking steps of the selected recipe into order, allowing the user to proceed with the cooking process by checking each step.

[2340] "Substitute material" refers to another material that can perform an equivalent function if a particular material is not available.

[2341] "Means for suggesting alternative materials" refers to a function that suggests suitable alternative materials from a database when a user has difficulty obtaining a particular material.

[2342] "Means for uploading and sharing photos of food" refers to the ability for users to upload photos of food they have cooked to the system and share them with other users.

[2343] "Means for displaying feedback and comments" refers to a function that displays ratings and opinions from other users on the screen so that users can check them.

[2344] "Means for searching and viewing restaurant menus" refers to the function of searching for menus offered at restaurants such as Japanese restaurants and displaying their contents.

[2345] "Cooking classes" refers to facilities and services that provide educational environments and programs for learning how to cook.

[2346] This invention is a system that allows users to effectively learn and practice Japanese cuisine at brick-and-mortar establishments such as Japanese restaurants and cooking schools. This system is primarily composed of a server, user terminals, and a user community. Each element is described in detail below.

[2347] 1. User Registration and Login

[2348] Program Generation

[2349] The server has an API endpoint for user registration and login. The user enters their "username," "email address," and "password" from their device and sends them to the server. The server receives this and stores the user information in a database. When the user logs in, they enter their "email address" and "password," which the server authenticates. After successful authentication, session information is generated and returned to the user.

[2350] Natural language explanations

[2351] In order for a user to use the system, they must first register. The user enters their "user name," "email address," and "password" on their terminal and sends a registration request to the server. The server receives this and saves the user information in a database. The "email address" and "password" entered by the user when logging in are compared with the information in the database for authentication, and if successful, session information is returned to the user. This completes user authentication.

[2352] 2. Search and browse recipes

[2353] Program Generation

[2354] When a user searches for a Japanese recipe, they enter a keyword (e.g., "sushi") into the device's search bar and send a search request to the server. The server extracts the relevant recipe information from the database and returns it to the device. The user can then select the recipe they are interested in from the displayed list of recipes and check its details.

[2355] Natural language explanations

[2356] When a user searches for Japanese cuisine, they enter specific keywords into the device's search bar and send a request to the server. The server then extracts the relevant recipe information from the database and sends it back to the device. The user can then select the recipe they are interested in from the displayed list and view detailed information.

[2357] 3. Step-by-step guide

[2358] Program Generation

[2359] The system displays the cooking steps for the recipe selected by the user, step by step. Specifically, the server receives the user's request, retrieves the corresponding step data from the database, and sends it to the device in sequence. The data may include photos and videos, and is displayed in a way that makes it easy for the user to understand.

[2360] Natural language explanations

[2361] A guide is provided to help users check the cooking steps for the recipe they have selected. Upon receiving a user's request, the server retrieves the corresponding step data from the database and sends it to the device in sequence. This allows the user to proceed with the cooking process while checking each step of the cooking procedure.

[2362] 4. Proposal of alternative materials

[2363] Program Generation

[2364] If a user has difficulty obtaining a particular ingredient, the user sends a request for a substitute ingredient from the device to the server, which retrieves the corresponding alternative from the database and returns it to the device. The user can then continue cooking using the suggested alternative.

[2365] Natural language explanations

[2366] If a user feels that a particular ingredient is difficult to obtain, they can send a request to the server for alternative ingredient suggestions. The server will then retrieve the relevant alternative ingredient suggestions from the database and send them back to the device, allowing the user to continue cooking using the suggested alternative ingredients.

[2367] 5. Food photo sharing and community support

[2368] Program Generation

[2369] After the dish is complete, the user takes a photo and uploads it to the server from their device along with their comments. The server stores the photo and comments in a database and updates the community feed, which also displays feedback and comments from other users.

[2370] Natural language explanations

[2371] Users take a photo of their finished dish and upload it to the server along with their comments from their device. The server stores the photo and comments in a database and updates the community feed, displaying feedback and comments from other users and encouraging interaction between users.

[2372] Specific examples

[2373] For example, consider a user who wants to make "sushi." After logging in, the user enters "sushi" in the search bar and presses the search button. The server retrieves recipes related to "sushi" from the database and returns the information to the device. The user selects "sushi" from the displayed list and checks the detailed information. Next, the user proceeds with the cooking process while following the step-by-step guide. If some of the ingredients are unavailable, the user can request alternative ingredients from the server and use the suggested substitute ingredients. Once the dish is complete, the user can upload photos and comments from the device to the server and share them with the community. The user can receive feedback and comments from other users and use them to improve their next cooking experience.

[2374] Prompt Sentence Examples

[2375] "A user launches the app and searches for 'sushi.' The app retrieves recipes related to 'sushi' from ...

Claims

1. a means for storing recipe data in a database; a means for authenticating a user; a means for performing a recipe search; a means of displaying step-by-step recipe instructions; a means of suggesting alternative materials; A way for users to upload and share photos of their food; a way to view feedback and comments from other users; A system including:

2. The system of claim 1 , further comprising means for storing user-uploaded food photos in a database and updating a community feed.

3. The system according to claim 1 , further comprising means for generating session information at the time of user authentication and managing the session information for each user.

Images