system

Abstract

Provide a system. A means for uploading a photo of an ingredient from a terminal to a server; a means for the server to analyze the uploaded photographs and generate an ingredients list; a means for the user to select a type of meal and number of days; a means for the server to suggest recipes based on the ingredient list and user selections; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's busy lifestyles, preparing meals healthily and efficiently without wasting ingredients is an important challenge for many people. However, typical recipe search and meal suggestion services cannot fully meet users' needs because they cannot consider the specific ingredients a user wants to use or their specific preferences (e.g., diet, reduced salt, etc.). They also lack mechanisms for providing real-time advice to users who are unsure about their cooking skills or presentation. Thus, the lack of a system that can suggest optimal recipes based on the ingredients a user has and provide feedback on individual preferences and the quality of the dish is a challenge. [Means for solving the problem]

[0005] The present invention provides a system that allows users to upload photos of ingredients from a device to a server, analyzes the ingredients, and generates an ingredient list. A means is provided for users to select the type of dish and the number of days the dish will be cooked, and the server proposes recipes based on the ingredient list and the user's selection. Additionally, a means is provided for premium members to generate recipes incorporating specific preferences (e.g., reduced salt, no oil, etc.), allowing for customization to meet user needs. Furthermore, a means is provided for users to upload photos of completed dishes, analyze the photos, and generate advice, realizing a system that provides real-time feedback on the cooking results. In this way, users can prepare optimal dishes based on their ingredients and preferences, helping them improve their cooking skills and presentation.

[0006] "Photo of ingredients" is image data of ingredients photographed by the user to be used in cooking.

[0007] "Terminal" refers to the device (e.g., smartphone, tablet, PC, etc.) that a user uses to take photos of ingredients and upload them to the server.

[0008] A "server" is a remote computer system capable of receiving ingredient photos, performing analysis, and generating an ingredient list.

[0009] The "ingredient list" is a list of ingredients that the user has, obtained by the server analyzing the photo.

[0010] "Cuisine type" refers to the style or theme of cuisine selected by the user (e.g., diet, hearty menu).

[0011] "Number of days" is a period for which the user can select how many days' worth of recipes they would like to receive.

[0012] "Recipe suggestion" means that the server suggests appropriate cooking methods and menus based on the ingredients list and user selections.

[0013] "Specific wishes" are special requests that the user wants reflected in the recipe suggestions (e.g., reduced salt, no oil).

[0014] A "premium member" is a user who can use additional services such as having specific requests reflected in recipes.

[0015] "Photo of finished dish" is image data of the dish taken by the user after the cooking is completed.

[0016] "Photo analysis" is the process by which the server analyzes uploaded images and evaluates the condition and presentation of the food.

[0017] "Advice" is feedback and suggestions for improvement generated by the server based on the results of photo analysis. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The system allows users to take photos of ingredients they have on hand, analyzes the photos, and suggests optimal recipes. The system also provides feedback on the user's individual preferences and the cooking results.

[0040] System Overview

[0041] This system is realized through the cooperation of the user's "device" and a "server." The device used by the user can be a smartphone, tablet, or PC, while the server is a remote computer system that performs image analysis and recipe suggestions.

[0042] composition

[0043] 1. Photograph and upload the material

[0044] The user takes a photo of the ingredients to be used in the dish using the camera on the device.

[0045] The terminal provides an interface for uploading the captured photo data to the server.

[0046] 2. Materials Analysis

[0047] The server receives the uploaded photos and uses image analysis algorithms to identify the ingredients.

[0048] The server lists the identified materials and stores them in a database.

[0049] 3. Food suggestions

[0050] The user selects the type of food they would like suggested (diet, hearty menu, etc.) and the period (number of days) on the device.

[0051] The terminal transmits the user's selection information to the server.

[0052] The server retrieves suitable recipes from a database based on the ingredient list and user selections.

[0053] The server provides the suggested recipes to the user's terminal.

[0054] 4. Premium recipe suggestions

[0055] Premium members can also enter specific preferences (such as low salt or no oil).

[0056] The server generates a customized recipe incorporating the user's specific wishes and sends it to the user's device.

[0057] 5. Food Photography Advice

[0058] The user takes a photo of the finished dish with the device and uploads it to the server.

[0059] The server analyzes the uploaded photos and generates advice on the appearance and quality of the dish.

[0060] The server provides the generated advice to the user's terminal.

[0061] Specific examples

[0062] For example, a user can take a photo of a day's worth of ingredients (chicken breast, broccoli, tomato, sweet potato) and upload it to the server. After this, the user can request a diet menu for the day. The server analyzes the ingredient list and suggests appropriate recipes for dieters, such as grilled chicken breast, steamed broccoli, tomato salad, and roasted sweet potato.

[0063] Additionally, if premium members add a preference for reduced salt intake, the server will provide recipes that are suitable for reduced salt intake (e.g., low-sodium grilled chicken breast, unsalted steamed broccoli, etc.).

[0064] Once the dish is complete, the user can upload a photo of it to the server, which can then analyze its appearance and color and provide advice on how to improve it, such as using a marinade to keep the food moist.

[0065] By following these steps, users can efficiently use their ingredients to create healthy and delicious meals.

[0066] The processing flow will be explained below.

[0067] Specific processing flow of the program

[0068] Photographing and uploading materials

[0069] Step 1:

[0070] The user takes a photo of the ingredients to be used in cooking using the device's camera.

[0071] Step 2:

[0072] Temporarily saves photos taken by the device.

[0073] Step 3:

[0074] The user uses the interface of the terminal to perform an operation to upload a photo of the material to the server.

[0075] Step 4:

[0076] The device sends the photo data to the specified API endpoint.

[0077] Step 5:

[0078] The server receives and stores the uploaded photo data.

[0079] Materials Analysis

[0080] Step 6:

[0081] The server runs an image analysis algorithm on the received photo data to identify the material.

[0082] Step 7:

[0083] The server lists the identified materials and stores them in a database.

[0084] Step 8:

[0085] The server transmits the generated material list to the terminal and notifies the user.

[0086] Cooking suggestions

[0087] Step 9:

[0088] The user selects the type of food and number of days for which they would like suggestions on their device.

[0089] Step 10:

[0090] The terminal transmits the user's selection information to the server.

[0091] Step 11:

[0092] The server searches the database for an appropriate recipe based on the ingredients list and the user's selections.

[0093] Step 12:

[0094] The server selects the most suitable recipe from the search results and generates a list of dishes.

[0095] Step 13:

[0096] The server sends the generated recipe list to the terminal and displays it to the user.

[0097] Premium recipe suggestions

[0098] Step 14:

[0099] A premium member user inputs their specific preferences (e.g., low salt, no oil) on the device.

[0100] Step 15:

[0101] The terminal transmits the desired conditions to the server.

[0102] Step 16:

[0103] The server searches the database for recipes that meet the desired conditions.

[0104] Step 17:

[0105] The server selects recipes that meet the conditions and generates a list of premium dishes.

[0106] Step 18:

[0107] The server transmits the generated premium dish list to the terminal and displays it to the user.

[0108] Food photography advice

[0109] Step 19:

[0110] The user takes a photo of the finished dish using the device.

[0111] Step 20:

[0112] The device sends the photo data to the specified API endpoint.

[0113] Step 21:

[0114] The server receives the uploaded photos and analyzes them.

[0115] Step 22:

[0116] The server generates advice based on the results of analyzing the photo.

[0117] Step 23:

[0118] The server sends the generated advice to the terminal and displays it to the user.

[0119] The above is the specific processing flow of the system and the operations performed at each step.

[0120] Example 1

[0121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] In today's busy lifestyles, users are required to maximize the use of ingredients on hand to create efficient and healthy meals. However, the process from selecting ingredients to searching for recipes and receiving feedback after cooking is cumbersome and time-consuming. It is also difficult to find recipes that meet specific nutritional restrictions or preferences. Therefore, there is a need for a system that can easily suggest healthy and varied meals using ingredients on hand, and also evaluate and improve the quality of the meals.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0124] In this invention, the server includes means for uploading photos of ingredients from a terminal to the data processing device, means for the data processing device to analyze the uploaded photos and generate an ingredient list, means for a user to select a type of dish and a cooking period, means for the data processing device to suggest a food preparation method based on the ingredient list and the user's selection, means for generating a customized food preparation method using a generative AI model based on the user's specific preferences, and means for analyzing the uploaded photos of the finished food and generating advice, thereby enabling users to efficiently use ingredients on hand to create healthy and diverse dishes and receive specific feedback on the results.

[0125] "Photo of ingredients" is image data of ingredients used in cooking that the user has photographed with the camera on the terminal.

[0126] A "terminal" is an electronic device used by a user, such as a smartphone, tablet, or PC.

[0127] A "data processing device" is a remote computer system such as a server or a cloud system.

[0128] The "material list" is data that lists the names of materials analyzed by the data processing device.

[0129] A "user" is an individual who uses this system to input ingredients and receive cooking suggestions.

[0130] "Type of food" is a specific food category selected by the user, such as diet menu or hearty menu.

[0131] The "period" is the number of days for which the user wishes to receive suggestions for selected dishes.

[0132] "Food preparation method" is a recipe suggested based on ingredients and conditions entered by the user.

[0133] A "generative AI model" is an artificial intelligence model that generates new data or text based on given information.

[0134] A "customized food recipe" is a recipe generated by a generative AI model that incorporates a user's specific preferences (e.g., low salt, no oil).

[0135] "Advice" is feedback that indicates improvements and suggestions for the appearance and finish of the dish based on the analysis results.

[0136] A "Premium Member" is a user who has special access rights to use certain additional features and services.

[0137] The present invention is a system that takes a photo of the ingredients a user has on hand, analyzes the photo, and suggests optimal recipes. The system also provides feedback on the user's individual preferences and the cooking results.

[0138] This system is realized through the cooperation of a server and a device used by the user. The device used by the user can be a smartphone, tablet, or PC, and the server is a remote computer system that performs image analysis and recipe suggestions.

[0139] Photographing and uploading materials

[0140] The user takes a photo of the ingredients to be used in the dish using the camera of the device, for example, using a camera app on a smartphone.

[0141] The device uploads the captured photo data to the server, which is sent over the Internet using an HTTP POST request.

[0142] Materials Analysis

[0143] The server receives the uploaded photos and identifies the materials using image analysis algorithms (e.g., TENSORFLOW® or OpenCV).

[0144] The server lists the identified materials and stores them in a database (e.g., MySQL (registered trademark) or PostgreSQL).

[0145] Cooking suggestions

[0146] The user selects the type of meal they want to be suggested (diet, hearty, etc.) and the period (number of days) on the device, for example, using a drop-down menu in the UI.

[0147] The terminal transmits the user's selection information to the server.

[0148] The server searches the database for suitable recipes based on the ingredients list and user selections, extracting data using SQL queries.

[0149] The server sends the search results in JSON format to the user's device, which receives them and displays them on the application screen.

[0150] Premium recipe suggestions

[0151] Premium users input their specific preferences, such as low salt or no oil, into the terminal application, for example using text fields or check boxes.

[0152] The terminal transmits this input information to the server.

[0153] The server uses a generative AI model (e.g., GPT-3®) to generate a customized recipe based on your specific preferences.

[0154] The server transmits the generated customized recipe to the user's terminal and displays it on the terminal application.

[0155] Food photography advice

[0156] Users take a photo of the finished dish with their smartphone camera and use the application to upload it to the server, where the photo is sent using an HTTP request.

[0157] The server analyzes the received photos and performs an evaluation using image analysis algorithms.

[0158] The server generates advice based on the evaluation results, including suggestions such as "marinate to keep the food moist" or "add parsley for color."

[0159] The server sends the generated advice as a text message to the user's terminal, which displays it.

[0160] By following these steps, users can efficiently use their ingredients to create healthy and delicious meals.

[0161] Examples of prompt statements

[0162] "Suggest a day's worth of meals suitable for today's diet"

[0163] "I'd like a low-sodium menu, so please tell me the recipe."

[0164] "Can you give me some advice on how to improve the appearance of this dish?"

[0165] This allows users to easily obtain recipes and advice that meet their needs.

[0166] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0167] Step 1:

[0168] The user takes a photo of the ingredients to be used in cooking using the camera app on their smartphone. The photo of the ingredients is obtained as input. The device displays the photo data on a preview screen within the app, and when the user is satisfied, they tap the "Upload" button. The photo data is sent to the server as output.

[0169] Step 2:

[0170] The device sends the photo data taken by the user to the server via the Internet. Specifically, it uses an HTTP POST request to send image data to the server. It receives the photo data as input and sends it to the server as output.

[0171] Step 3:

[0172] The server stores the photo data received from the device and applies image analysis algorithms (e.g., TensorFlow or OpenCV) to identify materials. The photo data is taken as input and a list of materials is generated through analysis. This list of materials is then generated as output and stored in a database.

[0173] Step 4:

[0174] The user selects the type of food they want to be recommended (for example, diet menu or hearty menu) and the period (number of days) on the terminal application. The user's selection information is obtained as input. The selection information is sent to the server as output.

[0175] Step 5:

[0176] The terminal transmits the user's selection information to the server. Specifically, the terminal transmits the selection information using an HTTP request. The terminal receives the user's selection information as input and transmits it to the server.

[0177] Step 6:

[0178] The server uses the ingredient list and user selection information to search the database for suitable recipes. Specifically, it uses SQL queries to extract data. The ingredient list and user selection information are received as input, and suitable recipe information is generated by performing a database search. This recipe information is generated as output.

[0179] Step 7:

[0180] The server sends the search results in JSON format to the user's device. It receives the generated recipe information as input and sends it to the device as JSON format data as output.

[0181] Step 8:

[0182] The device receives the recipe information and displays it on the application screen. It receives recipe information in JSON format as input and displays it on the application screen as output.

[0183] Step 9:

[0184] Premium members input their specific preferences, such as low salt or no oil, into the terminal application. The specific preferences are obtained as input. The preferences are sent to the server as output.

[0185] Step 10:

[0186] The terminal sends this input information to the server. Specifically, it sends the desired information using an HTTP request. It receives specific desired information as input and sends it to the server.

[0187] Step 11:

[0188] The server generates a customized recipe using a generative AI model (e.g., GPT-3) based on the user's specific preferences. The input is the user's preferences, and the output is the customized recipe information.

[0189] Step 12:

[0190] The server sends the generated customized recipe to the user's terminal and displays it in the terminal application, which receives the customized recipe information as input and sends it to the terminal as output.

[0191] Step 13:

[0192] The user takes a photo of the finished dish with their smartphone camera and uploads it to the server using the application. Similarly, the photo is sent using an HTTP request, which takes the photo of the finished dish as input and sends it to the server as output.

[0193] Step 14:

[0194] The server analyzes the received photos and uses image analysis algorithms to generate a rating, taking a photo of the finished dish as input and generating a rating result as output.

[0195] Step 15:

[0196] The server generates advice based on the evaluation results and suggests useful methods and improvements to the user.The evaluation results are taken as input and advice is generated as output.

[0197] Step 16:

[0198] The server sends this advice as a text message to the user's terminal, which displays it.The advice generated is taken as input and sent to the terminal as output.

[0199] (Application example 1)

[0200] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0201] Conventional ingredient analysis and recipe suggestion systems lacked a simple way for users to order additional ingredients when they needed more than they could use with the ingredients they already had on hand. They also struggled to provide customized recipes that accommodate users' dietary restrictions or specific preferences, leaving the user experience open to improvement. Furthermore, they lacked support for improving the quality of their cooking, as they had limited means of receiving feedback on the appearance and finish of the finished dish.

[0202] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0203] In this invention, the server includes a means for uploading photos of ingredients from a terminal to the server, a means for the server to analyze the uploaded photos and generate an ingredient list, a means for the user to select a type of dish and the number of days for cooking, a means for the server to suggest a recipe based on the ingredient list and the user's selection, a means for displaying a list of additional ingredients for the suggested recipe, and a means for ordering missing ingredients via a food delivery service. This allows users to efficiently utilize the ingredients they have on hand and order optimal recipes and missing ingredients for delivery all at once. Premium members can also receive customized recipes incorporating their specific preferences and feedback on the appearance and quality of the finished dish, thereby improving the user experience.

[0204] The "means for uploading a photo of an ingredient from a terminal to a server" is a function that allows a user to take a photo of an ingredient using his or her own terminal and transmit the data to a remote server.

[0205] "Means for the server to analyze uploaded photos and generate an ingredient list" refers to a function that uses an image analysis algorithm to analyze uploaded photo data on the server side, identify the ingredients in the photos, and create a list.

[0206] "Means for users to select the type of dish and number of days" refers to an interface that allows users to input or select the type of dish they want to make (e.g., diet, nutritional balance, etc.) and the period (number of days) for which they want to receive recipe suggestions.

[0207] "Means for the server to suggest recipes based on the ingredient list and user selection" refers to a function in which the server uses a database or generative AI model to generate and suggest optimal cooking recipes based on the generated ingredient list and user selection.

[0208] "Means for displaying a list of additional ingredients for a proposed recipe" refers to an interface that displays a list of additional ingredients that the user does not currently have based on the proposed recipe, allowing the user to visually confirm them.

[0209] The "means for ordering missing ingredients via a food delivery service" refers to a system for selecting the missing ingredients from the list, ordering those ingredients via the corresponding food delivery service, and arranging for delivery.

[0210] The embodiment of this invention is a system that proposes optimal recipes based on ingredients on hand and allows users to order additional ingredients via a food delivery service. This system includes the following main hardware and software components:

[0211] Hardware and software used

[0212] Smartphone (user device)

[0213] Cloud server (image analysis and recipe suggestions)

[0214] Image analysis software (OpenCV)

[0215] Machine learning models (TensorFlow and PyTorch)

[0216] Database (PostgreSQL, MongoDB)

[0217] Food delivery API (e.g. Uber Eats, DoorDash)

[0218] Generative AI models (e.g., GPT-3)

[0219] Specific explanation of the system

[0220] 1. Take and upload a photo of the material

[0221] Users take photos of materials they have on hand using their smartphones. The device then uploads the captured photo data to a cloud server. Users can use features such as laser focus assistance and brightness adjustment filters to improve analysis accuracy.

[0222] 2. Materials Analysis

[0223] The cloud server analyzes the uploaded photo data using image analysis software (OpenCV) to identify the materials and stores the results in a database as a text list. The analysis process uses object recognition algorithms and pre-trained models for material identification.

[0224] 3. Recipe suggestions and additional ingredients list

[0225] The server generates a recipe using a generative AI model (e.g., GPT-3) based on the generated ingredient list and the user's selected cooking type and number of days. The system also has a function to list additional ingredients that the user does not have in the recipe, allowing the user to visually check any missing ingredients.

[0226] 4. Food delivery orders

[0227] Users can check what ingredients they are missing on their device and order what they need via the food delivery service's API. The process includes real-time delivery time and cost estimates, and payment is processed using common electronic payment services.

[0228] 5. Food Photography Advice

[0229] Users can then upload photos of their completed dishes from their devices to the server, which uses image analysis software and generative AI models to generate feedback on the appearance and color of the dishes, improving the user experience.

[0230] Adding specific examples

[0231] Example prompts to input to the generative AI model

[0232] Prompt sentences from photo analysis results

[0233] The following ingredients were analyzed: chicken breast, broccoli, and tomato. Please suggest a diet recipe using these ingredients. The user wants to reduce sodium.

[0234] Food photography advice prompt

[0235] A user-uploaded food photo. Please provide feedback on the appearance and color of this dish.

[0236] This system allows users to efficiently utilize the ingredients they have on hand, create optimal recipes, and order missing ingredients for delivery all at once. Premium members can also receive customized recipes incorporating their specific preferences, as well as feedback on the appearance and quality of the finished dish, improving the user experience.

[0237] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0238] Step 1:

[0239] The user takes a photo of the material they have on hand using the device. As input, the user takes a photo of the material using the smartphone camera. This photo data is saved in the device application. It is desirable that the photo taken is high resolution.

[0240] Step 2:

[0241] The device uploads the captured photo data to the cloud server. The input is the photo data captured in step 1, and the output is the photo data uploaded to the cloud server. The upload process is securely performed via an internet connection.

[0242] Step 3:

[0243] The server receives the uploaded photos and analyzes the materials using image analysis software (OpenCV). The input is the uploaded photo data, and the output is a list of identified materials. The server saves the analysis results in a database in text format. The image analysis algorithm automatically identifies the materials in the photos.

[0244] Step 4:

[0245] The user selects the type of meal to be proposed (e.g., diet, regular menu, etc.) and the number of days on the terminal. The input is the information selected by the user on the terminal, and the output is the selection information being sent to the server. This includes operations on the user interface.

[0246] Step 5:

[0247] The server generates an optimal recipe using a generative AI model (e.g., GPT-3) based on the input ingredient list and user selection information. The input is the ingredient list and user selection information, and the output is a suggested recipe. The server inputs a prompt statement to the generative AI model to obtain the recipe.

[0248] Example prompt sentence:

[0249] The following ingredients were analyzed: chicken breast, broccoli, and tomato. Please suggest a diet recipe using these ingredients. The user wants to reduce sodium.

[0250] Step 6:

[0251] The server displays a list of additional ingredients that the user does not have based on the generated recipe. The input is the generated recipe, and the output is a list of missing ingredients. The list of missing ingredients is visually presented through a user interface.

[0252] Step 7:

[0253] The user selects the missing ingredients on the device and places an order via the food delivery service's API. The input is the list of missing ingredients and the user's selection. The output is order confirmation and delivery information. Payment processing is also included.

[0254] Step 8:

[0255] The user then takes a photo of the completed dish and uploads it to the cloud server. The input is the photo data of the completed dish, and the output is the photo data uploaded to the server.

[0256] Step 9:

[0257] The server analyzes the uploaded photos of the food using image analysis software and generative AI models to provide feedback on the appearance and color of the food. The input is the uploaded photo of the food, and the output is the feedback.

[0258] Example prompt sentence:

[0259] A user-uploaded food photo. Please provide feedback on the appearance and color of this dish.

[0260] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0261] This system allows users to take photos of ingredients they have on hand, analyzes the photos, and suggests optimal recipes. This system also provides feedback on the user's individual preferences and the cooking results. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized service.

[0262] System Overview

[0263] This system is realized through the cooperation of the user's "device," "server," and "emotion engine." The device used by the user can be a smartphone, tablet, or PC, and the server is a remote computer system that performs image analysis and recipe suggestions. The emotion engine is software or hardware that analyzes and reflects the user's emotions.

[0264] composition

[0265] 1. Photograph and upload the material

[0266] The user takes a photo of the ingredients to be used in the dish using the camera on the device.

[0267] The terminal provides an interface for uploading the captured photo data to the server.

[0268] 2. Materials Analysis

[0269] The server receives the uploaded photos and uses image analysis algorithms to identify the ingredients.

[0270] The server lists the identified materials and stores them in a database.

[0271] 3. Food suggestions

[0272] The user selects the type of food they would like suggested (diet, hearty menu, etc.) and the period (number of days) on the device.

[0273] The terminal transmits the user's selection information to the server.

[0274] The server searches the database for suitable recipes based on the ingredient list and user selection information.

[0275] The emotion engine analyzes the user's emotions and reflects them in recipe suggestions.

[0276] The server provides the suggested recipes to the user's terminal.

[0277] 4. Premium recipe suggestions

[0278] Premium members can also enter specific preferences (such as low salt or no oil).

[0279] The terminal transmits the desired conditions to the server.

[0280] The server generates a customized recipe incorporating the user's specific wishes and sends it to the user's device.

[0281] 5. Food Photography Advice

[0282] The user takes a photo of the finished dish with the device and uploads it to the server.

[0283] The server analyzes the uploaded photos and generates advice on the appearance and quality of the dish.

[0284] The emotion engine analyzes the user's emotions and reflects them in the advice.

[0285] The server provides the generated advice to the user's terminal.

[0286] Specific examples

[0287] For example, a user can take a photo of a day's worth of ingredients (chicken breast, broccoli, tomato, and sweet potato) and upload it to the server. After this, the user can request a diet menu for that day. The server analyzes the ingredients list, and the emotion engine analyzes the user's emotional state. If the server detects that the current mood is stressful, it will suggest recipes that have a stress-reducing effect. For example, it might suggest grilled chicken breast, steamed broccoli, tomato salad, and roasted sweet potato.

[0288] Furthermore, if a premium member user requests to reduce their salt intake, the server will provide low-salt recipes (e.g., low-salt grilled chicken breast, unsalted steamed broccoli, etc.). The emotion engine reflects the user's health concerns and prioritizes low-carb and low-calorie recipes.

[0289] After the dish is complete, the user uploads a photo of it to the server, which analyzes its appearance and color. The emotion engine then analyzes the user's emotional state and, if the user feels, for example, "I'm happy that it turned out well," provides advice on how to further improve the cooking technique (e.g., "Next time, it will look better if you balance the colors a little more").

[0290] By following these steps, users can efficiently use their own ingredients and create healthy and delicious meals. In addition, by analyzing and reflecting the user's emotions, they can receive more personalized service.

[0291] The processing flow will be explained below.

[0292] Specific processing flow of the program (system combining emotion engine)

[0293] Photographing and uploading materials

[0294] Step 1:

[0295] The user takes a photo of the ingredients to be used in cooking using the device's camera.

[0296] Step 2:

[0297] Temporarily saves photos taken by the device.

[0298] Step 3:

[0299] The user uses the interface of the terminal to perform an operation to upload a photo of the material to the server.

[0300] Step 4:

[0301] The device sends the photo data to the specified API endpoint.

[0302] Step 5:

[0303] The server receives and stores the uploaded photo data.

[0304] Materials Analysis

[0305] Step 6:

[0306] The server runs an image analysis algorithm on the received photo data to identify the material.

[0307] Step 7:

[0308] The server lists the identified materials and stores them in a database.

[0309] Step 8:

[0310] The server transmits the generated material list to the terminal and notifies the user.

[0311] Cooking suggestions

[0312] Step 9:

[0313] The user selects the type of food and number of days for which they would like suggestions on their device.

[0314] Step 10:

[0315] The terminal transmits the user's selection information to the server.

[0316] Step 11:

[0317] The server searches the database for an appropriate recipe based on the ingredients list and the user's selections.

[0318] Step 12:

[0319] The emotion engine analyzes the user's emotions and reflects them in recipe suggestions.

[0320] Step 13:

[0321] The server selects the most suitable recipes from the search results and the analysis results of the emotion engine, and generates a list of dishes.

[0322] Step 14:

[0323] The server sends the generated recipe list to the terminal and displays it to the user.

[0324] Premium recipe suggestions

[0325] Step 15:

[0326] A premium member user inputs their specific preferences (e.g., low salt, no oil) on the device.

[0327] Step 16:

[0328] The terminal transmits the desired conditions to the server.

[0329] Step 17:

[0330] The server searches the database for recipes that meet the desired conditions.

[0331] Step 18:

[0332] The emotion engine analyzes the user's emotions and reflects them in customized recipe suggestions.

[0333] Step 19:

[0334] The server selects recipes that meet the conditions and take into account the analysis results of the emotion engine, and generates a list of premium dishes.

[0335] Step 20:

[0336] The server transmits the generated premium dish list to the terminal and displays it to the user.

[0337] Food photography advice

[0338] Step 21:

[0339] The user takes a photo of the finished dish using the device.

[0340] Step 22:

[0341] The device sends the photo data to the specified API endpoint.

[0342] Step 23:

[0343] The server receives the uploaded photos and analyzes them.

[0344] Step 24:

[0345] The emotion engine analyzes the user's emotions and reflects them in the advice.

[0346] Step 25:

[0347] The server generates advice based on the results of the photo analysis and the emotion engine analysis.

[0348] Step 26:

[0349] The server sends the generated advice to the terminal and displays it to the user.

[0350] The above is the specific processing flow of the system that combines the emotion engine and the operations that are performed at each step.

[0351] Example 2

[0352] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0353] Conventional recipe suggestion systems lack the ability to provide personalized recipe suggestions based on the ingredients a user has available, and are unable to take into account the user's individual preferences and emotional state. Furthermore, there are no systems that provide feedback on the appearance or quality of a dish after it is completed.

[0354] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for uploading photos of ingredients from a terminal to the server, a means for the server to analyze the uploaded photos and generate an ingredient list, a means for the user to select a type of dish and a cooking period, a means for the server to suggest a recipe based on the ingredient list and the user's selection information, and a means for analyzing the user's emotional state using an emotion analysis means and reflecting the result in a recipe suggestion. This makes it possible to suggest personalized recipes based on the user's ingredients and provide feedback that takes the user's emotional state into consideration.

[0355] A "terminal" is a device that a user uses to take photos and upload data to a server, and includes a smartphone, tablet, PC, etc.

[0356] The "server" is a remote computer system that analyzes the photo data received from the terminal and generates ingredient lists and recipes.

[0357] "Image analysis algorithm" refers to the technology that recognizes and lists specific objects and materials from photographic data.

[0358] The "Ingredient List" is a list of ingredients obtained by analyzing photos uploaded from the terminal.

[0359] "User selection information" refers to information such as the type of food and the period selected by the user on the terminal.

[0360] "Emotion analysis means" refers to technology that analyzes the user's emotional state and reflects that in recipe suggestions and feedback.

[0361] A "recipe suggestion" is a recipe creation procedure generated by the server based on an ingredient list and user selections.

[0362] A "premium member" is a user who has the right to customize recipes by adding specific desired conditions.

[0363] "Feedback" is advice on the appearance and quality of the dish provided based on a photo of the finished dish.

[0364] System Overview

[0365] The present invention is a system that allows a user to take a photo of ingredients they have on hand, analyzes the photo, and suggests optimal recipes. Furthermore, it provides feedback on the user's individual preferences and the cooking results, and by using an emotion engine, it can provide a personalized service that responds to the user's emotions. This system mainly uses the following hardware and software:

[0366] Devices: Smartphones, tablets, computers

[0367] Server: A remote computer system (e.g., a cloud service)

[0368] Emotion Engine: Emotion Analysis Software

[0369] Program processing

[0370] The system's program includes a series of processes such as photographing and uploading ingredients, analyzing ingredients, suggesting dishes, suggesting premium recipes, and providing advice on food photography.

[0371] Photographing and uploading materials

[0372] The user takes a photo of the material at hand with the device's camera, and the device then uploads the photo to the server. This process includes an interface where the user can click a "Select Photo" button to select the photo they have taken.

[0373] Materials Analysis

[0374] The server receives the uploaded photo data and identifies the materials using image analysis algorithms (e.g., TensorFlow or OpenCV). The identified materials are listed and stored in a database.

[0375] Cooking suggestions

[0376] The user selects the type of food they want to be recommended (e.g., diet, hearty menu) and the period (number of days) on their device. The device sends the user's selection information to the server, which then suggests the optimal recipe based on the ingredients list and the user's selection information. In addition, the emotion engine analyzes the user's emotional state and reflects it in the recipe suggestions.

[0377] Premium recipe suggestions

[0378] Premium members can add specific preferences (e.g., less salt, no oil), and the device sends the preferences to the server, which then generates a customized recipe that takes the specific preferences into account.

[0379] Food photography advice

[0380] After cooking is complete, the user takes a photo of the finished dish and uploads it to the server. The server analyzes the photo and generates advice on the appearance and finish of the dish. The emotion engine analyzes the user's emotional state and reflects it in the advice.

[0381] Specific examples

[0382] For example, suppose a user has chicken breast, broccoli, tomato, and sweet potato. They take a photo of these ingredients and upload it to the server. When the user requests a daily diet menu, the server analyzes the ingredients. If the emotion engine determines that the user's emotional state is stressful, it will suggest recipes that have a stress-reducing effect. For example, grilled chicken breast, steamed broccoli, tomato salad, and roasted sweet potato may be suggested.

[0383] Additionally, if a premium member user requests reduced salt intake, the server will provide low-salt recipes (e.g., low-salt grilled chicken breast, unsalted steamed broccoli). The emotion engine reflects health-consciousness and prioritizes low-carb and low-calorie recipes. After the dish is complete, the user can upload a photo of it to the server, which will analyze its appearance and color and provide advice such as "Next time, you should try to balance the colors a little more" based on the user's emotion, such as "I'm happy that it looks good."

[0384] Prompt Sentence Examples

[0385] Here are some example prompts to input to a generative AI model:

[0386] Suggest a daily diet menu based on the ingredients the user has on hand (chicken breast, broccoli, tomato, sweet potato). Also, consider the user's current emotional state as stressful, and suggest recipes that will help reduce stress.

[0387] This allows users to efficiently use their ingredients and enjoy healthy cooking that is in line with their emotions.

[0388] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0389] Step 1: Photograph and upload your materials

[0390] Input: The user takes a photo of the ingredients at hand using the device's camera.

[0391] What it does: The user launches the camera application, clicks the "Take Photo" button to take a photo of the ingredient, and then selects the saved photo using the "Select Photo" function within the app.

[0392] Output: The selected photo data will be saved on your device.

[0393] Input: The device displays an interface for uploading the saved photo data to the server.

[0394] Specific behavior: The user clicks the "Upload Photo" button and the photo data is sent to the server.

[0395] Output: Uploaded photo data is sent to the server.

[0396] Step 2: Analyze the material

[0397] Input: The server receives the photo data sent from the device.

[0398] What happens: The server processes the HTTP request, decodes the photo data, and saves it.

[0399] Output: The saved photo data is passed to the image analysis module.

[0400] Input: The server uses image analysis algorithms (e.g. TensorFlow or OpenCV) to identify materials in the photo.

[0401] Specific operation: The server inputs the photo data into the image analysis model to identify and extract the material name.

[0402] Output: The identified materials are listed and stored in a database.

[0403] Step 3: Food suggestions

[0404] Input: The user selects the type of meal (e.g. diet, hearty) and the period (number of days) for which they would like suggestions on the device.

[0405] Specific behavior: The user makes a selection using a drop-down menu or radio buttons. After making a selection, the user clicks the "Submit" button.

[0406] Output: The selected data is sent from the device to the server.

[0407] Input: The server searches its database for a suitable recipe based on the ingredients list and user selections.

[0408] What it does: Executes an SQL query to extract the corresponding recipes from the database.

[0409] Output: The extracted recipe data is passed to the sentiment analysis module.

[0410] Input: The emotion engine analyzes the user's emotional state and reflects it in recipe suggestions.

[0411] What it does: The emotion engine analyzes the user's text input and past data to assess their current emotional state.

[0412] Output: The sentiment analysis results are added to the recipe data to generate the final recipe.

[0413] Input: The server sends the optimal recipe to the user's device.

[0414] Specific operation: Sends an HTTP response containing recipe data to the device.

[0415] Output: The user receives the suggested recipe on their device.

[0416] Step 4: Premium Recipe Proposal

[0417] Input: Premium members can additionally input specific preferences (e.g., low salt, no oil).

[0418] Specific actions: Enter your desired criteria using the text fields or checkboxes and click the "Submit" button.

[0419] Output: The desired condition data is sent from the terminal to the server.

[0420] Input: The server generates a customized recipe taking into account your desired conditions.

[0421] What it does: Adjusts SQL queries based on desired criteria to search and extract recipe data.

[0422] Output: The customized recipe data is sent to the user's device.

[0423] Step 5: Food Photography Tips

[0424] Input: The user takes a photo of the finished dish on their device and uploads it to the server.

[0425] Specific operations: Click the "Upload a photo of the finished dish" button, select a photo, and submit it.

[0426] Output: Photo data of the finished dish is sent to the server.

[0427] Input: The server receives the uploaded photo data.

[0428] Specific operation: Decodes and saves photo data.

[0429] Output: The saved photo data is passed to the image analysis module.

[0430] Input: The server analyzes the photo data and generates recommendations on the appearance and finish of the dish.

[0431] What it does: The image analysis model evaluates the photo and generates a feedback statement.

[0432] Output: The generated advice is passed to the sentiment analysis module.

[0433] Input: The emotion engine analyzes the user's emotional state and reflects it in the advice.

[0434] Specific operation: Integrate the sentiment analysis results into the feedback sentence.

[0435] Output: A feedback statement is generated for the user to receive.

[0436] (Application example 2)

[0437] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0438] Conventional recipe recommendation systems have been unable to flexibly suggest recipes based on the user's mood or physical condition, or when the user has a variety of cooking ingredients. There is a growing demand for recipe suggestions that take into account the passenger's emotions and current health status, especially in moving environments such as autonomous vehicles. However, no systems have been available to address this demand. This creates a need for personalized recipe suggestions that efficiently use ingredients carried by the user while also responding to the passenger's emotions and health status.

[0439] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading photos of ingredients from a terminal to the server, means for the server to analyze the uploaded photos and generate an ingredient list, means for the user to select the type of dish and the number of days, means for the server to suggest a recipe based on the ingredient list and the user's selection, means for a passenger to photograph and upload ingredients for the dish using a terminal in the autonomous vehicle, means for an emotion engine to analyze the passenger's emotions and reflect them in recipe suggestions, means for the server to suggest dishes that have effects such as stress reduction based on the passenger's current emotional state, means for generating recipes that incorporate specific requests for premium members, and means for the user to additionally input desired conditions (such as carbohydrate restriction or reduced salt). This makes it possible to efficiently use the ingredients that passengers have on hand while in an autonomous vehicle and to suggest personalized recipes that are optimal for the passenger's emotions and physical condition.

[0440] "Ingredients" refer to the ingredients and raw materials used to create a dish.

[0441] "Terminal" refers to electronic devices such as smartphones, tablets, and personal computers, which are devices that are directly operated by users.

[0442] A "server" is a computer system installed in the cloud or at a remote location, and is a device that analyzes and processes data.

[0443] "Upload" means sending data from the user's terminal to the server.

[0444] "Photo" refers to image data taken by a user using the camera function of the terminal.

[0445] "Image analysis" is a method by which the server analyzes photographic data to identify materials and other elements.

[0446] The "ingredient list" is a list of cooking ingredients identified through image analysis.

[0447] A "recipe" is a detailed description of the steps and ingredients needed to prepare a particular dish.

[0448] An "emotion engine" is a software or hardware system for analyzing a user's emotional state.

[0449] "Selection information" refers to conditions such as the type of food desired by the user and the number of days.

[0450] A "Premium Member" is a registered user who has access to additional services and customization features.

[0451] "Desired conditions" are requirements that the user specifically specifies when proposing a recipe (e.g., carbohydrate restriction, reduced salt, etc.).

[0452] An "autonomous vehicle" is a vehicle that operates automatically without a human driver.

[0453] "Passenger" means a user riding in an automated driving vehicle.

[0454] "Personalization" refers to customizing something to suit the preferences and needs of an individual user.

[0455] "Feedback" refers to the reactions and opinions of users regarding the services and suggestions provided.

[0456] The system of this invention allows users to photograph and upload ingredients they have, analyze them, and suggest optimal recipes. Furthermore, by combining it with an emotion engine, it is possible to suggest personalized recipes based on the user's emotional state and physical condition. Passengers in autonomous vehicles can receive cooking suggestions and advice tailored to their individual preferences using ingredients they carry with them while traveling.

[0457] Hardware and software used

[0458] Hardware

[0459] Smartphone (ANDROID (registered trademark), iOS)

[0460] Information systems in autonomous vehicles

[0461] Cloud server (AWS (registered trademark), GCP, etc.)

[0462] software

[0463] Image analysis algorithms (TensorFlow, OpenCV)

[0464] Emotion engine (IBM Watson®, Microsoft® Azure® Cognitive Services)

[0465] Recipe database (MySQL, PostgreSQL)

[0466] Mobile app development framework (Flutter (registered trademark), React Native)

[0467] System Operation

[0468] The user uses a smartphone to take a photo of the ingredients they are holding, and the photo data is uploaded to a cloud server via the autonomous vehicle's in-vehicle information system. The cloud server then uses image analysis algorithms to identify the ingredients in the photo and generate an ingredients list.

[0469] The user then selects the type of dish (e.g., diet menu, hearty menu, etc.) and the number of days (e.g., 1-day supply, 3-day supply), and sends this information from the device to the server. The server combines the selection information sent by the user with the ingredient list to search for the optimal recipe, and the emotion engine analyzes the user's emotional state and reflects it in the recipe suggestions.

[0470] For premium members, users can also enter specific preferences (e.g., low sugar, low salt), and this information is also sent to the server. The server then generates and provides customized recipes to users that take these preferences into account.

[0471] Once the dish is ready, the user takes a photo of the dish with their smartphone and uploads it to the server, which analyzes the photo and generates advice about the appearance and finish.The emotion engine analyzes the user's emotional state and reflects it in the advice.

[0472] Specific examples

[0473] For example, a passenger in an autonomous vehicle can take a photo of the chicken breast, tomatoes, and broccoli they purchased that day and upload it with their smartphone. At the same time, they can select a "diet menu" and input that they are currently feeling "stressed." The server analyzes the ingredient list, and the emotion engine suggests dishes that are effective in "reducing stress" (e.g., grilled chicken breast with a balanced salad).

[0474] Prompt sentence to input to the generative AI model:

[0475] I took photos of ingredients and uploaded them. I'd like a diet menu this time. I'm also feeling stressed right now. Please suggest recipes that take stress reduction into consideration.

[0476] The system allows passengers on the move to make the most of the ingredients they have on hand and easily create healthy and delicious meals tailored to their individual desires and emotional state.

[0477] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0478] Step 1:

[0479] The user uses a smartphone to take a photo of the ingredients to be used in cooking, launches the smartphone's camera app, and presses the capture button to generate photo data.

[0480] Input: smartphone, materials

[0481] Output: Photo data

[0482] Step 2:

[0483] The device uploads the captured photo data to the cloud server, and the device's application sends the photo data via the cloud service's API.

[0484] Input: Photo data

[0485] Output: Upload photo data to cloud server

[0486] Step 3:

[0487] The server receives the uploaded photos and uses image analysis algorithms to identify materials in the photos. The server performs image analysis using TensorFlow or OpenCV.

[0488] Input: Photo data

[0489] Output: Materials list

[0490] Step 4:

[0491] The server generates a material list and saves it in the database. The analyzed material information is saved in list format.

[0492] Input: Material data

[0493] Output: Material list saved in database

[0494] Step 5:

[0495] The user selects the type of food and the number of days they want to eat it using the device application. The user taps on the selection screen to select an item, and the selection information is generated.

[0496] Input: User selection information (food type, number of days)

[0497] Output: Selection information

[0498] Step 6:

[0499] The device sends the selection information to the cloud server. The device application calls an API to send the selection information to the server.

[0500] Input: Selection information

[0501] Output: Send to server

[0502] Step 7:

[0503] The server searches the database for the best recipe based on the ingredients list and selection information. It queries the database using an SQL query.

[0504] Input: Material list, selection information

[0505] Output: Recipe candidates

[0506] Step 8:

[0507] The emotion engine analyzes the user's emotional state and reflects it in recipe suggestions. The emotion engine obtains emotional data from the user's voice and facial expressions, analyzes it, and reflects it in recipe search results.

[0508] Input: Emotion data

[0509] Output: Personalized recipe suggestions

[0510] Step 9:

[0511] The server provides personalized recipes to the user's device, where the suggested recipes are sent to an application on the device and displayed on the screen.

[0512] Input: Personalized Recipe

[0513] Output: Recipe display

[0514] Step 10:

[0515] For premium members, users can enter additional specific conditions they desire (e.g., carbohydrate restriction, reduced salt intake, etc.). Desired condition data is generated when users enter additional desired conditions into the terminal.

[0516] Input: Desired conditions

[0517] Output: Desired condition data

[0518] Step 11:

[0519] The device sends the desired conditions data to the server. The device application calls the API to send the desired conditions data to the server.

[0520] Input: Desired conditions data

[0521] Output: Send to server

[0522] Step 12:

[0523] The server generates a customized recipe based on the user's desired conditions and provides it to the user's device. The server then searches the database again, selects a recipe that meets the desired conditions, and sends it to the device.

[0524] Input: Desired conditions, material list

[0525] Output: Customized recipe

[0526] Step 13:

[0527] Once the user has completed the cooking, they take a photo of it with their smartphone and upload it to the server. They then use the smartphone's camera app to take another photo and upload it to the cloud server.

[0528] Input: Food photo

[0529] Output: Upload photos to cloud server

[0530] Step 14:

[0531] The server analyzes the uploaded photos of the finished dish, and the emotion engine analyzes the user's emotional state and reflects it in the advice. The server uses an image analysis algorithm, and the emotion engine analyzes the user's emotional data.

[0532] Input: Photos of finished dishes, user emotion data

[0533] Output: Advice

[0534] Step 15:

[0535] The server provides the generated advice to the user's terminal, where it is sent to an application on the terminal and displayed to the user.

[0536] Input: Advice

[0537] Output: Advice display

[0538] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0539] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0540] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0541] [Second embodiment]

[0542] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0543] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0544] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0545] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0546] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0547] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0548] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0549] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0550] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0551] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0552] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0553] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0554] The system allows users to take photos of ingredients they have on hand, analyzes the photos, and suggests optimal recipes. The system also provides feedback on the user's individual preferences and the cooking results.

[0555] System Overview

[0556] This system is realized through the cooperation of the user's "device" and a "server." The device used by the user can be a smartphone, tablet, or PC, while the server is a remote computer system that performs image analysis and recipe suggestions.

[0557] composition

[0558] 1. Photograph and upload the material

[0559] The user takes a photo of the ingredients to be used in the dish using the camera on the device.

[0560] The terminal provides an interface for uploading the captured photo data to the server.

[0561] 2. Materials Analysis

[0562] The server receives the uploaded photos and uses image analysis algorithms to identify the ingredients.

[0563] The server lists the identified materials and stores them in a database.

[0564] 3. Food suggestions

[0565] The user selects the type of food they would like suggested (diet, hearty menu, etc.) and the period (number of days) on the device.

[0566] The terminal transmits the user's selection information to the server.

[0567] The server retrieves suitable recipes from a database based on the ingredient list and user selections.

[0568] The server provides the suggested recipes to the user's terminal.

[0569] 4. Premium recipe suggestions

[0570] Premium members can also enter specific preferences (such as low salt or no oil).

[0571] The server generates a customized recipe incorporating the user's specific wishes and sends it to the user's device.

[0572] 5. Food Photography Advice

[0573] The user takes a photo of the finished dish with the device and uploads it to the server.

[0574] The server analyzes the uploaded photos and generates advice on the appearance and quality of the dish.

[0575] The server provides the generated advice to the user's terminal.

[0576] Specific examples

[0577] For example, a user can take a photo of a day's worth of ingredients (chicken breast, broccoli, tomato, sweet potato) and upload it to the server. After this, the user can request a diet menu for the day. The server analyzes the ingredient list and suggests appropriate recipes for dieters, such as grilled chicken breast, steamed broccoli, tomato salad, and roasted sweet potato.

[0578] Additionally, if premium members add a preference for reduced salt intake, the server will provide recipes that are suitable for reduced salt intake (e.g., low-sodium grilled chicken breast, unsalted steamed broccoli, etc.).

[0579] Once the dish is complete, the user can upload a photo of it to the server, which can then analyze its appearance and color and provide advice on how to improve it, such as using a marinade to keep the food moist.

[0580] By following these steps, users can efficiently use their ingredients to create healthy and delicious meals.

[0581] The processing flow will be explained below.

[0582] Specific processing flow of the program

[0583] Photographing and uploading materials

[0584] Step 1:

[0585] The user takes a photo of the ingredients to be used in cooking using the device's camera.

[0586] Step 2:

[0587] Temporarily saves photos taken by the device.

[0588] Step 3:

[0589] The user uses the interface of the terminal to perform an operation to upload a photo of the material to the server.

[0590] Step 4:

[0591] The device sends the photo data to the specified API endpoint.

[0592] Step 5:

[0593] The server receives and stores the uploaded photo data.

[0594] Materials Analysis

[0595] Step 6:

[0596] The server runs an image analysis algorithm on the received photo data to identify the material.

[0597] Step 7:

[0598] The server lists the identified materials and stores them in a database.

[0599] Step 8:

[0600] The server transmits the generated material list to the terminal and notifies the user.

[0601] Cooking suggestions

[0602] Step 9:

[0603] The user selects the type of food and number of days for which they would like suggestions on their device.

[0604] Step 10:

[0605] The terminal transmits the user's selection information to the server.

[0606] Step 11:

[0607] The server searches the database for an appropriate recipe based on the ingredients list and the user's selections.

[0608] Step 12:

[0609] The server selects the most suitable recipe from the search results and generates a list of dishes.

[0610] Step 13:

[0611] The server sends the generated recipe list to the terminal and displays it to the user.

[0612] Premium recipe suggestions

[0613] Step 14:

[0614] A premium member user inputs their specific preferences (e.g., low salt, no oil) on the device.

[0615] Step 15:

[0616] The terminal transmits the desired conditions to the server.

[0617] Step 16:

[0618] The server searches the database for recipes that meet the desired conditions.

[0619] Step 17:

[0620] The server selects recipes that meet the conditions and generates a list of premium dishes.

[0621] Step 18:

[0622] The server transmits the generated premium dish list to the terminal and displays it to the user.

[0623] Food photography advice

[0624] Step 19:

[0625] The user takes a photo of the finished dish using the device.

[0626] Step 20:

[0627] The device sends the photo data to the specified API endpoint.

[0628] Step 21:

[0629] The server receives the uploaded photos and analyzes them.

[0630] Step 22:

[0631] The server generates advice based on the results of analyzing the photo.

[0632] Step 23:

[0633] The server sends the generated advice to the terminal and displays it to the user.

[0634] The above is the specific processing flow of the system and the operations performed at each step.

[0635] Example 1

[0636] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0637] In today's busy lifestyles, users are required to maximize the use of ingredients on hand to create efficient and healthy meals. However, the process from selecting ingredients to searching for recipes and receiving feedback after cooking is cumbersome and time-consuming. It is also difficult to find recipes that meet specific nutritional restrictions or preferences. Therefore, there is a need for a system that can easily suggest healthy and varied meals using ingredients on hand, and also evaluate and improve the quality of the meals.

[0638] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0639] In this invention, the server includes means for uploading photos of ingredients from a terminal to the data processing device, means for the data processing device to analyze the uploaded photos and generate an ingredient list, means for a user to select a type of dish and a cooking period, means for the data processing device to suggest a food preparation method based on the ingredient list and the user's selection, means for generating a customized food preparation method using a generative AI model based on the user's specific preferences, and means for analyzing the uploaded photos of the finished food and generating advice, thereby enabling users to efficiently use ingredients on hand to create healthy and diverse dishes and receive specific feedback on the results.

[0640] "Photo of ingredients" is image data of ingredients used in cooking that the user has photographed with the camera on the terminal.

[0641] A "terminal" is an electronic device used by a user, such as a smartphone, tablet, or PC.

[0642] A "data processing device" is a remote computer system such as a server or a cloud system.

[0643] The "material list" is data that lists the names of materials analyzed by the data processing device.

[0644] A "user" is an individual who uses this system to input ingredients and receive cooking suggestions.

[0645] "Type of food" is a specific food category selected by the user, such as diet menu or hearty menu.

[0646] The "period" is the number of days for which the user wishes to receive suggestions for selected dishes.

[0647] "Food preparation method" is a recipe suggested based on ingredients and conditions entered by the user.

[0648] A "generative AI model" is an artificial intelligence model that generates new data or text based on given information.

[0649] A "customized food recipe" is a recipe generated by a generative AI model that incorporates a user's specific preferences (e.g., low salt, no oil).

[0650] "Advice" is feedback that indicates improvements and suggestions for the appearance and finish of the dish based on the analysis results.

[0651] A "Premium Member" is a user who has special access rights to use certain additional features and services.

[0652] The present invention is a system that takes a photo of the ingredients a user has on hand, analyzes the photo, and suggests optimal recipes. The system also provides feedback on the user's individual preferences and the cooking results.

[0653] This system is realized through the cooperation of a server and a device used by the user. The device used by the user can be a smartphone, tablet, or PC, and the server is a remote computer system that performs image analysis and recipe suggestions.

[0654] Photographing and uploading materials

[0655] The user takes a photo of the ingredients to be used in the dish using the camera of the device, for example, using a camera app on a smartphone.

[0656] The device uploads the captured photo data to the server, which is sent over the Internet using an HTTP POST request.

[0657] Materials Analysis

[0658] The server receives the uploaded photos and identifies the materials using image analysis algorithms (e.g., TensorFlow or OpenCV).

[0659] The server lists the identified materials and stores them in a database (e.g., MySQL or PostgreSQL).

[0660] Cooking suggestions

[0661] The user selects the type of meal they want to be suggested (diet, hearty, etc.) and the period (number of days) on the device, for example, using a drop-down menu in the UI.

[0662] The terminal transmits the user's selection information to the server.

[0663] The server searches the database for suitable recipes based on the ingredients list and user selections, extracting data using SQL queries.

[0664] The server sends the search results in JSON format to the user's device, which receives them and displays them on the application screen.

[0665] Premium recipe suggestions

[0666] Premium users input their specific preferences, such as low salt or no oil, into the terminal application, for example using text fields or check boxes.

[0667] The terminal transmits this input information to the server.

[0668] The server uses a generative AI model (e.g., GPT-3) to generate a customized recipe based on your specific preferences.

[0669] The server transmits the generated customized recipe to the user's terminal and displays it on the terminal application.

[0670] Food photography advice

[0671] Users take a photo of the finished dish with their smartphone camera and use the application to upload it to the server, where the photo is sent using an HTTP request.

[0672] The server analyzes the received photos and performs an evaluation using image analysis algorithms.

[0673] The server generates advice based on the evaluation results, including suggestions such as "marinate to keep the food moist" or "add parsley for color."

[0674] The server sends the generated advice as a text message to the user's terminal, which displays it.

[0675] By following these steps, users can efficiently use their ingredients to create healthy and delicious meals.

[0676] Examples of prompt statements

[0677] "Suggest a day's worth of meals suitable for today's diet"

[0678] "I'd like a low-sodium menu, so please tell me the recipe."

[0679] "Can you give me some advice on how to improve the appearance of this dish?"

[0680] This allows users to easily obtain recipes and advice that meet their needs.

[0681] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0682] Step 1:

[0683] The user takes a photo of the ingredients to be used in cooking using the camera app on their smartphone. The photo of the ingredients is obtained as input. The device displays the photo data on a preview screen within the app, and when the user is satisfied, they tap the "Upload" button. The photo data is sent to the server as output.

[0684] Step 2:

[0685] The device sends the photo data taken by the user to the server via the Internet. Specifically, it uses an HTTP POST request to send image data to the server. It receives the photo data as input and sends it to the server as output.

[0686] Step 3:

[0687] The server stores the photo data received from the device and applies image analysis algorithms (e.g., TensorFlow or OpenCV) to identify materials. The photo data is taken as input and a list of materials is generated through analysis. This list of materials is then generated as output and stored in a database.

[0688] Step 4:

[0689] The user selects the type of food they want to be recommended (for example, diet menu or hearty menu) and the period (number of days) on the terminal application. The user's selection information is obtained as input. The selection information is sent to the server as output.

[0690] Step 5:

[0691] The terminal transmits the user's selection information to the server. Specifically, the terminal transmits the selection information using an HTTP request. The terminal receives the user's selection information as input and transmits it to the server.

[0692] Step 6:

[0693] The server uses the ingredient list and user selection information to search the database for suitable recipes. Specifically, it uses SQL queries to extract data. The ingredient list and user selection information are received as input, and suitable recipe information is generated by performing a database search. This recipe information is generated as output.

[0694] Step 7:

[0695] The server sends the search results in JSON format to the user's device. It receives the generated recipe information as input and sends it to the device as JSON format data as output.

[0696] Step 8:

[0697] The device displays the received recipe information on the application screen. It receives recipe information in JSON format as input and displays it on the application screen as output.

[0698] Step 9:

[0699] Premium members input their specific preferences, such as low salt or no oil, into the terminal application. The specific preferences are obtained as input. The preferences are sent to the server as output.

[0700] Step 10:

[0701] The terminal sends this input information to the server. Specifically, it sends the desired information using an HTTP request. It receives specific desired information as input and sends it to the server.

[0702] Step 11:

[0703] The server generates a customized recipe using a generative AI model (e.g., GPT-3) based on the user's specific preferences. The input is the user's preferences, and the output is the customized recipe information.

[0704] Step 12:

[0705] The server sends the generated customized recipe to the user's terminal and displays it in the terminal application, which receives the customized recipe information as input and sends it to the terminal as output.

[0706] Step 13:

[0707] The user takes a photo of the finished dish with their smartphone camera and uploads it to the server using the application. Similarly, the photo is sent using an HTTP request, which takes the photo of the finished dish as input and sends it to the server as output.

[0708] Step 14:

[0709] The server analyzes the received photos and uses image analysis algorithms to generate a rating, taking a photo of the finished dish as input and generating a rating result as output.

[0710] Step 15:

[0711] The server generates advice based on the evaluation results and suggests useful methods and improvements to the user.The evaluation results are taken as input and advice is generated as output.

[0712] Step 16:

[0713] The server sends this advice as a text message to the user's terminal, which displays it.The advice generated is taken as input and sent to the terminal as output.

[0714] (Application example 1)

[0715] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0716] Conventional ingredient analysis and recipe suggestion systems lacked a simple way for users to order additional ingredients when they needed more than they could use with the ingredients they already had on hand. They also struggled to provide customized recipes that accommodate users' dietary restrictions or specific preferences, leaving the user experience open to improvement. Furthermore, they lacked support for improving the quality of their cooking, as they had limited means of receiving feedback on the appearance and finish of the finished dish.

[0717] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0718] In this invention, the server includes a means for uploading photos of ingredients from a terminal to the server, a means for the server to analyze the uploaded photos and generate an ingredient list, a means for the user to select a type of dish and the number of days for cooking, a means for the server to suggest a recipe based on the ingredient list and the user's selection, a means for displaying a list of additional ingredients for the suggested recipe, and a means for ordering missing ingredients via a food delivery service. This allows users to efficiently utilize the ingredients they have on hand and order optimal recipes and missing ingredients for delivery all at once. Premium members can also receive customized recipes incorporating their specific preferences and feedback on the appearance and quality of the finished dish, thereby improving the user experience.

[0719] The "means for uploading a photo of an ingredient from a terminal to a server" is a function that allows a user to take a photo of an ingredient using his or her own terminal and transmit the data to a remote server.

[0720] "Means for the server to analyze uploaded photos and generate an ingredient list" refers to a function that uses an image analysis algorithm to analyze uploaded photo data on the server side, identify the ingredients in the photos, and create a list.

[0721] "Means for users to select the type of dish and number of days" refers to an interface that allows users to input or select the type of dish they want to make (e.g., diet, nutritional balance, etc.) and the period (number of days) for which they want to receive recipe suggestions.

[0722] "Means for the server to suggest recipes based on the ingredient list and user selection" refers to a function in which the server uses a database or generative AI model to generate and suggest optimal cooking recipes based on the generated ingredient list and user selection.

[0723] "Means for displaying a list of additional ingredients for a proposed recipe" refers to an interface that displays a list of additional ingredients that the user does not currently have based on the proposed recipe, allowing the user to visually confirm them.

[0724] The "means for ordering missing ingredients via a food delivery service" refers to a system for selecting the missing ingredients from the list, ordering those ingredients via the corresponding food delivery service, and arranging for delivery.

[0725] The embodiment of this invention is a system that proposes optimal recipes based on ingredients on hand and allows users to order additional ingredients via a food delivery service. This system includes the following main hardware and software components:

[0726] Hardware and software used

[0727] Smartphone (user device)

[0728] Cloud server (image analysis and recipe suggestions)

[0729] Image analysis software (OpenCV)

[0730] Machine learning models (TensorFlow and PyTorch)

[0731] Database (PostgreSQL, MongoDB)

[0732] Food delivery API (e.g. Uber Eats, DoorDash)

[0733] Generative AI models (e.g., GPT-3)

[0734] Specific explanation of the system

[0735] 1. Take and upload a photo of the material

[0736] Users take photos of materials they have on hand using their smartphones. The device then uploads the captured photo data to a cloud server. Users can use features such as laser focus assistance and brightness adjustment filters to improve analysis accuracy.

[0737] 2. Materials Analysis

[0738] The cloud server analyzes the uploaded photo data using image analysis software (OpenCV) to identify the materials and stores the results in a database as a text list. The analysis process uses object recognition algorithms and pre-trained models for material identification.

[0739] 3. Recipe suggestions and additional ingredients list

[0740] The server generates a recipe using a generative AI model (e.g., GPT-3) based on the generated ingredient list and the user's selected cooking type and number of days. The system also has a function to list additional ingredients that the user does not have in the recipe, allowing the user to visually check any missing ingredients.

[0741] 4. Food delivery orders

[0742] Users can check what ingredients they are missing on their device and order what they need via the food delivery service's API. The process includes real-time delivery time and cost estimates, and payment is processed using common electronic payment services.

[0743] 5. Food Photography Advice

[0744] Users can then upload photos of their completed dishes from their devices to the server, which uses image analysis software and generative AI models to generate feedback on the appearance and color of the dishes, improving the user experience.

[0745] Adding specific examples

[0746] Example prompts to input to the generative AI model

[0747] Prompt sentences from photo analysis results

[0748] The following ingredients were analyzed: chicken breast, broccoli, and tomato. Please suggest a diet recipe using these ingredients. The user wants to reduce sodium.

[0749] Food photography advice prompt

[0750] A user-uploaded food photo. Please provide feedback on the appearance and color of this dish.

[0751] This system allows users to efficiently utilize the ingredients they have on hand, create optimal recipes, and order missing ingredients for delivery all at once. Premium members can also receive customized recipes incorporating their specific preferences, as well as feedback on the appearance and quality of the finished dish, improving the user experience.

[0752] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0753] Step 1:

[0754] The user takes a photo of the material they have on hand using the device. As input, the user takes a photo of the material using the smartphone camera. This photo data is saved in the device application. It is desirable that the photo taken is high resolution.

[0755] Step 2:

[0756] The device uploads the captured photo data to the cloud server. The input is the photo data captured in step 1, and the output is the photo data uploaded to the cloud server. The upload process is securely performed via an internet connection.

[0757] Step 3:

[0758] The server receives the uploaded photos and analyzes the materials using image analysis software (OpenCV). The input is the uploaded photo data, and the output is a list of identified materials. The server saves the analysis results in a database in text format. The image analysis algorithm automatically identifies the materials in the photos.

[0759] Step 4:

[0760] The user selects the type of meal to be proposed (e.g., diet, regular menu, etc.) and the number of days on the terminal. The input is the information selected by the user on the terminal, and the output is the selection information being sent to the server. This includes operations on the user interface.

[0761] Step 5:

[0762] The server generates an optimal recipe using a generative AI model (e.g., GPT-3) based on the input ingredient list and user selection information. The input is the ingredient list and user selection information, and the output is a suggested recipe. The server inputs a prompt statement to the generative AI model to obtain the recipe.

[0763] Example prompt sentence:

[0764] The following ingredients were analyzed: chicken breast, broccoli, and tomato. Please suggest a diet recipe using these ingredients. The user wants to reduce sodium.

[0765] Step 6:

[0766] The server displays a list of additional ingredients that the user does not have based on the generated recipe. The input is the generated recipe, and the output is a list of missing ingredients. The list of missing ingredients is visually presented through a user interface.

[0767] Step 7:

[0768] The user selects the missing ingredients on the device and places an order via the food delivery service's API. The input is the list of missing ingredients and the user's selection. The output is order confirmation and delivery information. Payment processing is also included.

[0769] Step 8:

[0770] The user then takes a photo of the completed dish and uploads it to the cloud server. The input is the photo data of the completed dish, and the output is the photo data uploaded to the server.

[0771] Step 9:

[0772] The server analyzes the uploaded photos of the food using image analysis software and generative AI models to provide feedback on the appearance and color of the food. The input is the uploaded photo of the food, and the output is the feedback.

[0773] Example prompt sentence:

[0774] A user-uploaded food photo. Please provide feedback on the appearance and color of this dish.

[0775] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0776] This system allows users to take photos of ingredients they have on hand, analyzes the photos, and suggests optimal recipes. This system also provides feedback on the user's individual preferences and the cooking results. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized service.

[0777] System Overview

[0778] This system is realized through the cooperation of the user's "device," "server," and "emotion engine." The device used by the user can be a smartphone, tablet, or PC, and the server is a remote computer system that performs image analysis and recipe suggestions. The emotion engine is software or hardware that analyzes and reflects the user's emotions.

[0779] composition

[0780] 1. Photograph and upload the material

[0781] The user takes a photo of the ingredients to be used in the dish using the camera on the device.

[0782] The terminal provides an interface for uploading the captured photo data to the server.

[0783] 2. Materials Analysis

[0784] The server receives the uploaded photos and uses image analysis algorithms to identify the ingredients.

[0785] The server lists the identified materials and stores them in a database.

[0786] 3. Food suggestions

[0787] The user selects the type of food they would like suggested (diet, hearty menu, etc.) and the period (number of days) on the device.

[0788] The terminal transmits the user's selection information to the server.

[0789] The server searches the database for suitable recipes based on the ingredient list and user selection information.

[0790] The emotion engine analyzes the user's emotions and reflects them in recipe suggestions.

[0791] The server provides the suggested recipes to the user's terminal.

[0792] 4. Premium recipe suggestions

[0793] Premium members can also enter specific preferences (such as low salt or no oil).

[0794] The terminal transmits the desired conditions to the server.

[0795] The server generates a customized recipe incorporating the user's specific wishes and sends it to the user's device.

[0796] 5. Food Photography Advice

[0797] The user takes a photo of the finished dish with the device and uploads it to the server.

[0798] The server analyzes the uploaded photos and generates advice on the appearance and quality of the dish.

[0799] The emotion engine analyzes the user's emotions and reflects them in the advice.

[0800] The server provides the generated advice to the user's terminal.

[0801] Specific examples

[0802] For example, a user can take a photo of a day's worth of ingredients (chicken breast, broccoli, tomato, and sweet potato) and upload it to the server. After this, the user can request a diet menu for that day. The server analyzes the ingredients list, and the emotion engine analyzes the user's emotional state. If the server detects that the current mood is stressful, it will suggest recipes that have a stress-reducing effect. For example, it might suggest grilled chicken breast, steamed broccoli, tomato salad, and roasted sweet potato.

[0803] Furthermore, if a premium member user requests to reduce their salt intake, the server will provide low-salt recipes (e.g., low-salt grilled chicken breast, unsalted steamed broccoli, etc.). The emotion engine reflects the user's health concerns and prioritizes low-carb and low-calorie recipes.

[0804] After the dish is complete, the user uploads a photo of it to the server, which analyzes its appearance and color. The emotion engine then analyzes the user's emotional state and, if the user feels, for example, "I'm happy that it turned out well," provides advice on how to further improve the cooking technique (e.g., "Next time, it will look better if you balance the colors a little more").

[0805] By following these steps, users can efficiently use their own ingredients and create healthy and delicious meals. In addition, by analyzing and reflecting the user's emotions, they can receive more personalized service.

[0806] The processing flow will be explained below.

[0807] Specific processing flow of the program (system combining emotion engine)

[0808] Photographing and uploading materials

[0809] Step 1:

[0810] The user takes a photo of the ingredients to be used in cooking using the device's camera.

[0811] Step 2:

[0812] Temporarily saves photos taken by the device.

[0813] Step 3:

[0814] The user uses the interface of the terminal to perform an operation to upload a photo of the material to the server.

[0815] Step 4:

[0816] The device sends the photo data to the specified API endpoint.

[0817] Step 5:

[0818] The server receives and stores the uploaded photo data.

[0819] Materials Analysis

[0820] Step 6:

[0821] The server runs an image analysis algorithm on the received photo data to identify the material.

[0822] Step 7:

[0823] The server lists the identified materials and stores them in a database.

[0824] Step 8:

[0825] The server transmits the generated material list to the terminal and notifies the user.

[0826] Cooking suggestions

[0827] Step 9:

[0828] The user selects the type of food and number of days for which they would like suggestions on their device.

[0829] Step 10:

[0830] The terminal transmits the user's selection information to the server.

[0831] Step 11:

[0832] The server searches the database for an appropriate recipe based on the ingredients list and the user's selections.

[0833] Step 12:

[0834] The emotion engine analyzes the user's emotions and reflects them in recipe suggestions.

[0835] Step 13:

[0836] The server selects the most suitable recipes from the search results and the analysis results of the emotion engine, and generates a list of dishes.

[0837] Step 14:

[0838] The server sends the generated recipe list to the terminal and displays it to the user.

[0839] Premium recipe suggestions

[0840] Step 15:

[0841] A premium member user inputs their specific preferences (e.g., low salt, no oil) on the device.

[0842] Step 16:

[0843] The terminal transmits the desired conditions to the server.

[0844] Step 17:

[0845] The server searches the database for recipes that meet the desired conditions.

[0846] Step 18:

[0847] The emotion engine analyzes the user's emotions and reflects them in customized recipe suggestions.

[0848] Step 19:

[0849] The server selects recipes that meet the conditions and take into account the analysis results of the emotion engine, and generates a list of premium dishes.

[0850] Step 20:

[0851] The server transmits the generated premium dish list to the terminal and displays it to the user.

[0852] Food photography advice

[0853] Step 21:

[0854] The user takes a photo of the finished dish using the device.

[0855] Step 22:

[0856] The device sends the photo data to the specified API endpoint.

[0857] Step 23:

[0858] The server receives the uploaded photos and analyzes them.

[0859] Step 24:

[0860] The emotion engine analyzes the user's emotions and reflects them in the advice.

[0861] Step 25:

[0862] The server generates advice based on the results of the photo analysis and the emotion engine analysis.

[0863] Step 26:

[0864] The server sends the generated advice to the terminal and displays it to the user.

[0865] The above is the specific processing flow of the system that combines the emotion engine and the operations that are performed at each step.

[0866] Example 2

[0867] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0868] Conventional recipe suggestion systems lack the ability to provide personalized recipe suggestions based on the ingredients a user has available, and are unable to take into account the user's individual preferences and emotional state. Furthermore, there are no systems that provide feedback on the appearance or quality of a dish after it is completed.

[0869] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for uploading photos of ingredients from a terminal to the server, a means for the server to analyze the uploaded photos and generate an ingredient list, a means for the user to select a type of dish and a cooking period, a means for the server to suggest a recipe based on the ingredient list and the user's selection information, and a means for analyzing the user's emotional state using an emotion analysis means and reflecting the result in a recipe suggestion. This makes it possible to suggest personalized recipes based on the user's ingredients and provide feedback that takes the user's emotional state into consideration.

[0870] A "terminal" is a device that a user uses to take photos and upload data to a server, and includes a smartphone, tablet, PC, etc.

[0871] The "server" is a remote computer system that analyzes the photo data received from the terminal and generates ingredient lists and recipes.

[0872] "Image analysis algorithm" refers to the technology that recognizes and lists specific objects and materials from photographic data.

[0873] The "Ingredient List" is a list of ingredients obtained by analyzing photos uploaded from the terminal.

[0874] "User selection information" refers to information such as the type of food and the period selected by the user on the terminal.

[0875] "Emotion analysis means" refers to technology that analyzes the user's emotional state and reflects that in recipe suggestions and feedback.

[0876] A "recipe suggestion" is a recipe creation procedure generated by the server based on an ingredient list and user selections.

[0877] A "premium member" is a user who has the right to customize recipes by adding specific desired conditions.

[0878] "Feedback" is advice on the appearance and quality of the dish provided based on a photo of the finished dish.

[0879] System Overview

[0880] The present invention is a system that allows a user to take a photo of ingredients they have on hand, analyzes the photo, and suggests optimal recipes. Furthermore, it provides feedback on the user's individual preferences and the cooking results, and by using an emotion engine, it can provide a personalized service that responds to the user's emotions. This system mainly uses the following hardware and software:

[0881] Devices: Smartphones, tablets, computers

[0882] Server: A remote computer system (e.g., a cloud service)

[0883] Emotion Engine: Emotion Analysis Software

[0884] Program processing

[0885] The system's program includes a series of processes such as photographing and uploading ingredients, analyzing ingredients, suggesting dishes, suggesting premium recipes, and providing advice on food photography.

[0886] Photographing and uploading materials

[0887] The user takes a photo of the material at hand with the device's camera, and the device then uploads the photo to the server. This process includes an interface where the user can click a "Select Photo" button to select the photo they have taken.

[0888] Materials Analysis

[0889] The server receives the uploaded photo data and identifies the materials using image analysis algorithms (e.g., TensorFlow or OpenCV). The identified materials are listed and stored in a database.

[0890] Cooking suggestions

[0891] The user selects the type of food they want to be recommended (e.g., diet, hearty menu) and the period (number of days) on their device. The device sends the user's selection information to the server, which then suggests the optimal recipe based on the ingredients list and the user's selection information. In addition, the emotion engine analyzes the user's emotional state and reflects it in the recipe suggestions.

[0892] Premium recipe suggestions

[0893] Premium members can add specific preferences (e.g., less salt, no oil), and the device sends the preferences to the server, which then generates a customized recipe that takes the specific preferences into account.

[0894] Food photography advice

[0895] After cooking is complete, the user takes a photo of the finished dish and uploads it to the server. The server analyzes the photo and generates advice on the appearance and finish of the dish. The emotion engine analyzes the user's emotional state and reflects it in the advice.

[0896] Specific examples

[0897] For example, suppose a user has chicken breast, broccoli, tomato, and sweet potato. They take a photo of these ingredients and upload it to the server. When the user requests a daily diet menu, the server analyzes the ingredients. If the emotion engine determines that the user's emotional state is stressful, it will suggest recipes that have a stress-reducing effect. For example, grilled chicken breast, steamed broccoli, tomato salad, and roasted sweet potato may be suggested.

[0898] Additionally, if a premium member user requests reduced salt intake, the server will provide low-salt recipes (e.g., low-salt grilled chicken breast, unsalted steamed broccoli). The emotion engine reflects health-consciousness and prioritizes low-carb and low-calorie recipes. After the dish is complete, the user can upload a photo of it to the server, which will analyze its appearance and color and provide advice such as "Next time, you should try to balance the colors a little more" based on the user's emotion, such as "I'm happy that it looks good."

[0899] Prompt Sentence Examples

[0900] Here are some example prompts to input to a generative AI model:

[0901] Suggest a daily diet menu based on the ingredients the user has on hand (chicken breast, broccoli, tomato, sweet potato). Also, consider the user's current emotional state as stressful, and suggest recipes that will help reduce stress.

[0902] This allows users to efficiently use their ingredients and enjoy healthy cooking that is in line with their emotions.

[0903] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0904] Step 1: Photograph and upload your materials

[0905] Input: The user takes a photo of the ingredients at hand using the device's camera.

[0906] What it does: The user launches the camera application, clicks the "Take Photo" button to take a photo of the ingredient, and then selects the saved photo using the "Select Photo" function within the app.

[0907] Output: The selected photo data will be saved on your device.

[0908] Input: The device displays an interface for uploading the saved photo data to the server.

[0909] Specific behavior: The user clicks the "Upload Photo" button and the photo data is sent to the server.

[0910] Output: Uploaded photo data is sent to the server.

[0911] Step 2: Analyze the material

[0912] Input: The server receives the photo data sent from the device.

[0913] What happens: The server processes the HTTP request, decodes the photo data, and saves it.

[0914] Output: The saved photo data is passed to the image analysis module.

[0915] Input: The server uses image analysis algorithms (e.g. TensorFlow or OpenCV) to identify materials in the photo.

[0916] Specific operation: The server inputs the photo data into the image analysis model to identify and extract the material name.

[0917] Output: The identified materials are listed and stored in a database.

[0918] Step 3: Food suggestions

[0919] Input: The user selects the type of meal (e.g. diet, hearty) and the period (number of days) for which they would like suggestions on the device.

[0920] Specific behavior: The user makes a selection using a drop-down menu or radio buttons. After making a selection, the user clicks the "Submit" button.

[0921] Output: The selected data is sent from the device to the server.

[0922] Input: The server searches its database for a suitable recipe based on the ingredients list and user selections.

[0923] What it does: Executes an SQL query to extract the corresponding recipes from the database.

[0924] Output: The extracted recipe data is passed to the sentiment analysis module.

[0925] Input: The emotion engine analyzes the user's emotional state and reflects it in recipe suggestions.

[0926] What it does: The emotion engine analyzes the user's text input and past data to assess their current emotional state.

[0927] Output: The sentiment analysis results are added to the recipe data to generate the final recipe.

[0928] Input: The server sends the optimal recipe to the user's device.

[0929] Specific operation: Sends an HTTP response containing recipe data to the device.

[0930] Output: The user receives the suggested recipe on their device.

[0931] Step 4: Premium Recipe Proposal

[0932] Input: Premium members can additionally input specific preferences (e.g., low salt, no oil).

[0933] Specific actions: Enter your desired criteria using the text fields or checkboxes and click the "Submit" button.

[0934] Output: The desired condition data is sent from the terminal to the server.

[0935] Input: The server generates a customized recipe taking into account your desired conditions.

[0936] What it does: Adjusts SQL queries based on desired criteria to search and extract recipe data.

[0937] Output: The customized recipe data is sent to the user's device.

[0938] Step 5: Food Photography Tips

[0939] Input: The user takes a photo of the finished dish on their device and uploads it to the server.

[0940] Specific operations: Click the "Upload a photo of the finished dish" button, select a photo, and submit it.

[0941] Output: Photo data of the finished dish is sent to the server.

[0942] Input: The server receives the uploaded photo data.

[0943] Specific operation: Decodes and saves photo data.

[0944] Output: The saved photo data is passed to the image analysis module.

[0945] Input: The server analyzes the photo data and generates recommendations on the appearance and finish of the dish.

[0946] What it does: The image analysis model evaluates the photo and generates a feedback statement.

[0947] Output: The generated advice is passed to the sentiment analysis module.

[0948] Input: The emotion engine analyzes the user's emotional state and reflects it in the advice.

[0949] Specific operation: Integrate the sentiment analysis results into the feedback sentence.

[0950] Output: A feedback statement is generated for the user to receive.

[0951] (Application example 2)

[0952] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0953] Conventional recipe recommendation systems have been unable to flexibly suggest recipes based on the user's mood or physical condition, or when the user has a variety of cooking ingredients. There is a growing demand for recipe suggestions that take into account the passenger's emotions and current health status, especially in moving environments such as autonomous vehicles. However, no systems have been available to address this demand. This creates a need for personalized recipe suggestions that efficiently use ingredients carried by the user while also responding to the passenger's emotions and health status.

[0954] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading photos of ingredients from a terminal to the server, means for the server to analyze the uploaded photos and generate an ingredient list, means for the user to select the type of dish and the number of days, means for the server to suggest a recipe based on the ingredient list and the user's selection, means for a passenger to photograph and upload ingredients for the dish using a terminal in the autonomous vehicle, means for an emotion engine to analyze the passenger's emotions and reflect them in recipe suggestions, means for the server to suggest dishes that have effects such as stress reduction based on the passenger's current emotional state, means for generating recipes that incorporate specific requests for premium members, and means for the user to additionally input desired conditions (such as carbohydrate restriction or reduced salt). This makes it possible to efficiently use the ingredients that passengers have on hand while in an autonomous vehicle and to suggest personalized recipes that are optimal for the passenger's emotions and physical condition.

[0955] "Ingredients" refer to the ingredients and raw materials used to create a dish.

[0956] "Terminal" refers to electronic devices such as smartphones, tablets, and personal computers, which are devices that are directly operated by users.

[0957] A "server" is a computer system installed in the cloud or at a remote location, and is a device that analyzes and processes data.

[0958] "Upload" means sending data from the user's terminal to the server.

[0959] "Photo" refers to image data taken by a user using the camera function of the terminal.

[0960] "Image analysis" is a method by which the server analyzes photographic data to identify materials and other elements.

[0961] The "ingredient list" is a list of cooking ingredients identified through image analysis.

[0962] A "recipe" is a detailed description of the steps and ingredients needed to prepare a particular dish.

[0963] An "emotion engine" is a software or hardware system for analyzing a user's emotional state.

[0964] "Selection information" refers to conditions such as the type of food desired by the user and the number of days.

[0965] A "Premium Member" is a registered user who has access to additional services and customization features.

[0966] "Desired conditions" are requirements that the user specifically specifies when proposing a recipe (e.g., carbohydrate restriction, reduced salt, etc.).

[0967] An "autonomous vehicle" is a vehicle that operates automatically without a human driver.

[0968] "Passenger" means a user riding in an automated driving vehicle.

[0969] "Personalization" refers to customizing something to suit the preferences and needs of an individual user.

[0970] "Feedback" refers to the reactions and opinions of users regarding the services and suggestions provided.

[0971] The system of this invention allows users to photograph and upload ingredients they have, analyze them, and suggest optimal recipes. Furthermore, by combining it with an emotion engine, it is possible to suggest personalized recipes based on the user's emotional state and physical condition. Passengers in autonomous vehicles can receive cooking suggestions and advice tailored to their individual preferences using ingredients they carry with them while traveling.

[0972] Hardware and software used

[0973] Hardware

[0974] Smartphone (Android, iOS)

[0975] Information systems in autonomous vehicles

[0976] Cloud server (AWS, GCP, etc.)

[0977] software

[0978] Image analysis algorithms (TensorFlow, OpenCV)

[0979] Emotion engine (IBM Watson, Microsoft Azure Cognitive Services)

[0980] Recipe database (MySQL, PostgreSQL)

[0981] Mobile app development frameworks (Flutter, React Native)

[0982] System Operation

[0983] The user uses a smartphone to take a photo of the ingredients they are holding, and the photo data is uploaded to a cloud server via the autonomous vehicle's in-vehicle information system. The cloud server then uses image analysis algorithms to identify the ingredients in the photo and generate an ingredients list.

[0984] The user then selects the type of dish (e.g., diet menu, hearty menu, etc.) and the number of days (e.g., 1-day supply, 3-day supply), and sends this information from the device to the server. The server combines the selection information sent by the user with the ingredient list to search for the optimal recipe, and the emotion engine analyzes the user's emotional state and reflects it in the recipe suggestions.

[0985] For premium members, users can also enter specific preferences (e.g., low sugar, low salt), and this information is also sent to the server. The server then generates and provides customized recipes to users that take these preferences into account.

[0986] Once the dish is ready, the user takes a photo of the dish with their smartphone and uploads it to the server, which analyzes the photo and generates advice about the appearance and finish.The emotion engine analyzes the user's emotional state and reflects it in the advice.

[0987] Specific examples

[0988] For example, a passenger in an autonomous vehicle can take a photo of the chicken breast, tomatoes, and broccoli they purchased that day and upload it with their smartphone. At the same time, they can select a "diet menu" and input that they are currently feeling "stressed." The server analyzes the ingredient list, and the emotion engine suggests dishes that are effective in "reducing stress" (e.g., grilled chicken breast with a balanced salad).

[0989] Prompt sentence to input to the generative AI model:

[0990] I took photos of ingredients and uploaded them. I'd like a diet menu this time. I'm also feeling stressed right now. Please suggest recipes that take stress reduction into consideration.

[0991] The system allows passengers on the move to make the most of the ingredients they have on hand and easily create healthy and delicious meals tailored to their individual desires and emotional state.

[0992] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0993] Step 1:

[0994] The user uses a smartphone to take a photo of the ingredients to be used in cooking, launches the smartphone's camera app, and presses the capture button to generate photo data.

[0995] Input: smartphone, materials

[0996] Output: Photo data

[0997] Step 2:

[0998] The device uploads the captured photo data to the cloud server, and the device's application sends the photo data via the cloud service's API.

[0999] Input: Photo data

[1000] Output: Upload photo data to cloud server

[1001] Step 3:

[1002] The server receives the uploaded photos and uses image analysis algorithms to identify materials in the photos. The server performs image analysis using TensorFlow or OpenCV.

[1003] Input: Photo data

[1004] Output: Materials list

[1005] Step 4:

[1006] The server generates a material list and saves it in the database. The analyzed material information is saved in list format.

[1007] Input: Material data

[1008] Output: Material list saved in database

[1009] Step 5:

[1010] The user selects the type of food and the number of days they want to eat it using the device application. The user taps on the selection screen to select an item, and the selection information is generated.

[1011] Input: User selection information (food type, number of days)

[1012] Output: Selection information

[1013] Step 6:

[1014] The device sends the selection information to the cloud server. The device application calls an API to send the selection information to the server.

[1015] Input: Selection information

[1016] Output: Send to server

[1017] Step 7:

[1018] The server searches the database for the best recipe based on the ingredients list and selection information. It queries the database using an SQL query.

[1019] Input: Material list, selection information

[1020] Output: Recipe candidates

[1021] Step 8:

[1022] The emotion engine analyzes the user's emotional state and reflects it in recipe suggestions. The emotion engine obtains emotional data from the user's voice and facial expressions, analyzes it, and reflects it in recipe search results.

[1023] Input: Emotion data

[1024] Output: Personalized recipe suggestions

[1025] Step 9:

[1026] The server provides personalized recipes to the user's device, where the suggested recipes are sent to an application on the device and displayed on the screen.

[1027] Input: Personalized Recipe

[1028] Output: Recipe display

[1029] Step 10:

[1030] For premium members, users can enter additional specific conditions they desire (e.g., carbohydrate restriction, reduced salt intake, etc.). Desired condition data is generated when users enter additional desired conditions into the terminal.

[1031] Input: Desired conditions

[1032] Output: Desired condition data

[1033] Step 11:

[1034] The device sends the desired conditions data to the server. The device application calls the API to send the desired conditions data to the server.

[1035] Input: Desired conditions data

[1036] Output: Send to server

[1037] Step 12:

[1038] The server generates a customized recipe based on the user's desired conditions and provides it to the user's device. The server then searches the database again, selects a recipe that meets the desired conditions, and sends it to the device.

[1039] Input: Desired conditions, material list

[1040] Output: Customized recipe

[1041] Step 13:

[1042] Once the user has completed the cooking, they take a photo of it with their smartphone and upload it to the server. They then use the smartphone's camera app to take another photo and upload it to the cloud server.

[1043] Input: Food photo

[1044] Output: Upload photos to cloud server

[1045] Step 14:

[1046] The server analyzes the uploaded photos of the finished dish, and the emotion engine analyzes the user's emotional state and reflects it in the advice. The server uses an image analysis algorithm, and the emotion engine analyzes the user's emotional data.

[1047] Input: Photos of finished dishes, user emotion data

[1048] Output: Advice

[1049] Step 15:

[1050] The server provides the generated advice to the user's terminal, where it is sent to an application on the terminal and displayed to the user.

[1051] Input: Advice

[1052] Output: Advice display

[1053] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1054] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1055] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1056] [Third embodiment]

[1057] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1058] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1059] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1060] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1061] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1062] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1063] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1064] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1065] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1066] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1067] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1068] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1069] The system allows users to take photos of ingredients they have on hand, analyzes the photos, and suggests optimal recipes. The system also provides feedback on the user's individual preferences and the cooking results.

[1070] System Overview

[1071] This system is realized through the cooperation of the user's "device" and a "server." The device used by the user can be a smartphone, tablet, or PC, while the server is a remote computer system that performs image analysis and recipe suggestions.

[1072] composition

[1073] 1. Photograph and upload the material

[1074] The user takes a photo of the ingredients to be used in the dish using the camera on the device.

[1075] The terminal provides an interface for uploading the captured photo data to the server.

[1076] 2. Materials Analysis

[1077] The server receives the uploaded photos and uses image analysis algorithms to identify the ingredients.

[1078] The server lists the identified materials and stores them in a database.

[1079] 3. Food suggestions

[1080] The user selects the type of food they would like suggested (diet, hearty menu, etc.) and the period (number of days) on the device.

[1081] The terminal transmits the user's selection information to the server.

[1082] The server retrieves suitable recipes from a database based on the ingredient list and user selections.

[1083] The server provides the suggested recipes to the user's terminal.

[1084] 4. Premium recipe suggestions

[1085] Premium members can also enter specific preferences (such as low salt or no oil).

[1086] The server generates a customized recipe incorporating the user's specific wishes and sends it to the user's device.

[1087] 5. Food Photography Advice

[1088] The user takes a photo of the finished dish with the device and uploads it to the server.

[1089] The server analyzes the uploaded photos and generates advice on the appearance and quality of the dish.

[1090] The server provides the generated advice to the user's terminal.

[1091] Specific examples

[1092] For example, a user can take a photo of a day's worth of ingredients (chicken breast, broccoli, tomato, sweet potato) and upload it to the server. After this, the user can request a diet menu for the day. The server analyzes the ingredient list and suggests appropriate recipes for dieters, such as grilled chicken breast, steamed broccoli, tomato salad, and roasted sweet potato.

[1093] Additionally, if premium members add a preference for reduced salt intake, the server will provide recipes that are suitable for reduced salt intake (e.g., low-sodium grilled chicken breast, unsalted steamed broccoli, etc.).

[1094] Once the dish is complete, the user can upload a photo of it to the server, which can then analyze its appearance and color and provide advice on how to improve it, such as using a marinade to keep the food moist.

[1095] By following these steps, users can efficiently use their ingredients to create healthy and delicious meals.

[1096] The processing flow will be explained below.

[1097] Specific processing flow of the program

[1098] Photographing and uploading materials

[1099] Step 1:

[1100] The user takes a photo of the ingredients to be used in cooking using the device's camera.

[1101] Step 2:

[1102] Temporarily saves photos taken by the device.

[1103] Step 3:

[1104] The user uses the interface of the terminal to perform an operation to upload a photo of the material to the server.

[1105] Step 4:

[1106] The device sends the photo data to the specified API endpoint.

[1107] Step 5:

[1108] The server receives and stores the uploaded photo data.

[1109] Materials Analysis

[1110] Step 6:

[1111] The server runs an image analysis algorithm on the received photo data to identify the material.

[1112] Step 7:

[1113] The server lists the identified materials and stores them in a database.

[1114] Step 8:

[1115] The server transmits the generated material list to the terminal and notifies the user.

[1116] Cooking suggestions

[1117] Step 9:

[1118] The user selects the type of food and number of days for which they would like suggestions on their device.

[1119] Step 10:

[1120] The terminal transmits the user's selection information to the server.

[1121] Step 11:

[1122] The server searches the database for an appropriate recipe based on the ingredients list and the user's selections.

[1123] Step 12:

[1124] The server selects the most suitable recipe from the search results and generates a list of dishes.

[1125] Step 13:

[1126] The server sends the generated recipe list to the terminal and displays it to the user.

[1127] Premium recipe suggestions

[1128] Step 14:

[1129] A premium member user inputs their specific preferences (e.g., low salt, no oil) on the device.

[1130] Step 15:

[1131] The terminal transmits the desired conditions to the server.

[1132] Step 16:

[1133] The server searches the database for recipes that meet the desired conditions.

[1134] Step 17:

[1135] The server selects recipes that meet the conditions and generates a list of premium dishes.

[1136] Step 18:

[1137] The server transmits the generated premium dish list to the terminal and displays it to the user.

[1138] Food photography advice

[1139] Step 19:

[1140] The user takes a photo of the finished dish using the device.

[1141] Step 20:

[1142] The device sends the photo data to the specified API endpoint.

[1143] Step 21:

[1144] The server receives the uploaded photos and analyzes them.

[1145] Step 22:

[1146] The server generates advice based on the results of analyzing the photo.

[1147] Step 23:

[1148] The server sends the generated advice to the terminal and displays it to the user.

[1149] The above is the specific processing flow of the system and the operations performed at each step.

[1150] Example 1

[1151] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1152] In today's busy lifestyles, users are required to maximize the use of ingredients on hand to create efficient and healthy meals. However, the process from selecting ingredients to searching for recipes and receiving feedback after cooking is cumbersome and time-consuming. It is also difficult to find recipes that meet specific nutritional restrictions or preferences. Therefore, there is a need for a system that can easily suggest healthy and varied meals using ingredients on hand, and also evaluate and improve the quality of the meals.

[1153] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1154] In this invention, the server includes means for uploading photos of ingredients from a terminal to the data processing device, means for the data processing device to analyze the uploaded photos and generate an ingredient list, means for a user to select a type of dish and a cooking period, means for the data processing device to suggest a food preparation method based on the ingredient list and the user's selection, means for generating a customized food preparation method using a generative AI model based on the user's specific preferences, and means for analyzing the uploaded photos of the finished food and generating advice, thereby enabling users to efficiently use ingredients on hand to create healthy and diverse dishes and receive specific feedback on the results.

[1155] "Photo of ingredients" is image data of ingredients used in cooking that the user has photographed with the camera on the terminal.

[1156] A "terminal" is an electronic device used by a user, such as a smartphone, tablet, or PC.

[1157] A "data processing device" is a remote computer system such as a server or a cloud system.

[1158] The "material list" is data that lists the names of materials analyzed by the data processing device.

[1159] A "user" is an individual who uses this system to input ingredients and receive cooking suggestions.

[1160] "Type of food" is a specific food category selected by the user, such as diet menu or hearty menu.

[1161] The "period" is the number of days for which the user wishes to receive suggestions for selected dishes.

[1162] "Food preparation method" is a recipe suggested based on ingredients and conditions entered by the user.

[1163] A "generative AI model" is an artificial intelligence model that generates new data or text based on given information.

[1164] A "customized food recipe" is a recipe generated by a generative AI model that incorporates a user's specific preferences (e.g., low salt, no oil).

[1165] "Advice" is feedback that indicates improvements and suggestions for the appearance and finish of the dish based on the analysis results.

[1166] A "Premium Member" is a user who has special access rights to use certain additional features and services.

[1167] The present invention is a system that takes a photo of the ingredients a user has on hand, analyzes the photo, and suggests optimal recipes. The system also provides feedback on the user's individual preferences and the cooking results.

[1168] This system is realized through the cooperation of a server and a device used by the user. The device used by the user can be a smartphone, tablet, or PC, and the server is a remote computer system that performs image analysis and recipe suggestions.

[1169] Photographing and uploading materials

[1170] The user takes a photo of the ingredients to be used in the dish using the camera of the device, for example, using a camera app on a smartphone.

[1171] The device uploads the captured photo data to the server, which is sent over the Internet using an HTTP POST request.

[1172] Materials Analysis

[1173] The server receives the uploaded photos and identifies the materials using image analysis algorithms (e.g., TensorFlow or OpenCV).

[1174] The server lists the identified materials and stores them in a database (e.g., MySQL or PostgreSQL).

[1175] Cooking suggestions

[1176] The user selects the type of meal they want to be suggested (diet, hearty, etc.) and the period (number of days) on the device, for example, using a drop-down menu in the UI.

[1177] The terminal transmits the user's selection information to the server.

[1178] The server searches the database for suitable recipes based on the ingredients list and user selections, extracting data using SQL queries.

[1179] The server sends the search results in JSON format to the user's device, which receives them and displays them on the application screen.

[1180] Premium recipe suggestions

[1181] Premium users input their specific preferences, such as low salt or no oil, into the terminal application, for example using text fields or check boxes.

[1182] The terminal transmits this input information to the server.

[1183] The server uses a generative AI model (e.g., GPT-3) to generate a customized recipe based on your specific preferences.

[1184] The server transmits the generated customized recipe to the user's terminal and displays it on the terminal application.

[1185] Food photography advice

[1186] Users take a photo of the finished dish with their smartphone camera and use the application to upload it to the server, where the photo is sent using an HTTP request.

[1187] The server analyzes the received photos and performs an evaluation using image analysis algorithms.

[1188] The server generates advice based on the evaluation results, including suggestions such as "marinate to keep the food moist" or "add parsley for color."

[1189] The server sends the generated advice as a text message to the user's terminal, which displays it.

[1190] By following these steps, users can efficiently use their ingredients to create healthy and delicious meals.

[1191] Examples of prompt statements

[1192] "Suggest a day's worth of meals suitable for today's diet"

[1193] "I'd like a low-sodium menu, so please tell me the recipe."

[1194] "Can you give me some advice on how to improve the appearance of this dish?"

[1195] This allows users to easily obtain recipes and advice that meet their needs.

[1196] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1197] Step 1:

[1198] The user takes a photo of the ingredients to be used in cooking using the camera app on their smartphone. The photo of the ingredients is obtained as input. The device displays the photo data on a preview screen within the app, and when the user is satisfied, they tap the "Upload" button. The photo data is sent to the server as output.

[1199] Step 2:

[1200] The device sends the photo data taken by the user to the server via the Internet. Specifically, it uses an HTTP POST request to send image data to the server. It receives the photo data as input and sends it to the server as output.

[1201] Step 3:

[1202] The server stores the photo data received from the device and applies image analysis algorithms (e.g., TensorFlow or OpenCV) to identify materials. The photo data is taken as input and a list of materials is generated through analysis. This list of materials is then generated as output and stored in a database.

[1203] Step 4:

[1204] The user selects the type of food they want to be recommended (for example, diet menu or hearty menu) and the period (number of days) on the terminal application. The user's selection information is obtained as input. The selection information is sent to the server as output.

[1205] Step 5:

[1206] The terminal transmits the user's selection information to the server. Specifically, the terminal transmits the selection information using an HTTP request. The terminal receives the user's selection information as input and transmits it to the server.

[1207] Step 6:

[1208] The server uses the ingredient list and user selection information to search the database for suitable recipes. Specifically, it uses SQL queries to extract data. The ingredient list and user selection information are received as input, and suitable recipe information is generated by performing a database search. This recipe information is generated as output.

[1209] Step 7:

[1210] The server sends the search results in JSON format to the user's device. It receives the generated recipe information as input and sends it to the device as JSON format data as output.

[1211] Step 8:

[1212] The device receives the recipe information and displays it on the application screen. It receives recipe information in JSON format as input and displays it on the application screen as output.

[1213] Step 9:

[1214] Premium members input their specific preferences, such as low salt or no oil, into the terminal application. The specific preferences are obtained as input. The preferences are sent to the server as output.

[1215] Step 10:

[1216] The terminal sends this input information to the server. Specifically, it sends the desired information using an HTTP request. It receives specific desired information as input and sends it to the server.

[1217] Step 11:

[1218] The server generates a customized recipe using a generative AI model (e.g., GPT-3) based on the user's specific preferences. The input is the user's preferences, and the output is the customized recipe information.

[1219] Step 12:

[1220] The server sends the generated customized recipe to the user's terminal and displays it in the terminal application, which receives the customized recipe information as input and sends it to the terminal as output.

[1221] Step 13:

[1222] The user takes a photo of the finished dish with their smartphone camera and uploads it to the server using the application. Similarly, the photo is sent using an HTTP request, which takes the photo of the finished dish as input and sends it to the server as output.

[1223] Step 14:

[1224] The server analyzes the received photos and uses image analysis algorithms to generate a rating, taking a photo of the finished dish as input and generating a rating result as output.

[1225] Step 15:

[1226] The server generates advice based on the evaluation results and suggests useful methods and improvements to the user.The evaluation results are taken as input and advice is generated as output.

[1227] Step 16:

[1228] The server sends this advice as a text message to the user's terminal, which displays it.The advice generated is taken as input and sent to the terminal as output.

[1229] (Application example 1)

[1230] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1231] Conventional ingredient analysis and recipe suggestion systems lacked a simple way for users to order additional ingredients when they needed more than they could use with the ingredients they already had on hand. They also struggled to provide customized recipes that accommodate users' dietary restrictions or specific preferences, leaving the user experience open to improvement. Furthermore, they lacked support for improving the quality of their cooking, as they had limited means of receiving feedback on the appearance and finish of the finished dish.

[1232] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1233] In this invention, the server includes a means for uploading photos of ingredients from a terminal to the server, a means for the server to analyze the uploaded photos and generate an ingredient list, a means for the user to select a type of dish and the number of days for cooking, a means for the server to suggest a recipe based on the ingredient list and the user's selection, a means for displaying a list of additional ingredients for the suggested recipe, and a means for ordering missing ingredients via a food delivery service. This allows users to efficiently utilize the ingredients they have on hand and order optimal recipes and missing ingredients for delivery all at once. Premium members can also receive customized recipes incorporating their specific preferences and feedback on the appearance and quality of the finished dish, thereby improving the user experience.

[1234] The "means for uploading a photo of an ingredient from a terminal to a server" is a function that allows a user to take a photo of an ingredient using his or her own terminal and transmit the data to a remote server.

[1235] "Means for the server to analyze uploaded photos and generate an ingredient list" refers to a function that uses an image analysis algorithm to analyze uploaded photo data on the server side, identify the ingredients in the photos, and create a list.

[1236] "Means for users to select the type of dish and number of days" refers to an interface that allows users to input or select the type of dish they want to make (e.g., diet, nutritional balance, etc.) and the period (number of days) for which they want to receive recipe suggestions.

[1237] "Means for the server to suggest recipes based on the ingredient list and user selection" refers to a function in which the server uses a database or generative AI model to generate and suggest optimal cooking recipes based on the generated ingredient list and user selection.

[1238] "Means for displaying a list of additional ingredients for a proposed recipe" refers to an interface that displays a list of additional ingredients that the user does not currently have based on the proposed recipe, allowing the user to visually confirm them.

[1239] The "means for ordering missing ingredients via a food delivery service" refers to a system for selecting the missing ingredients from the list, ordering those ingredients via the corresponding food delivery service, and arranging for delivery.

[1240] The embodiment of this invention is a system that proposes optimal recipes based on ingredients on hand and allows users to order additional ingredients via a food delivery service. This system includes the following main hardware and software components:

[1241] Hardware and software used

[1242] Smartphone (user device)

[1243] Cloud server (image analysis and recipe suggestions)

[1244] Image analysis software (OpenCV)

[1245] Machine learning models (TensorFlow and PyTorch)

[1246] Database (PostgreSQL, MongoDB)

[1247] Food delivery API (e.g. Uber Eats, DoorDash)

[1248] Generative AI models (e.g., GPT-3)

[1249] Specific explanation of the system

[1250] 1. Take and upload a photo of the material

[1251] Users take photos of materials they have on hand using their smartphones. The device then uploads the captured photo data to a cloud server. Users can use features such as laser focus assistance and brightness adjustment filters to improve analysis accuracy.

[1252] 2. Materials Analysis

[1253] The cloud server analyzes the uploaded photo data using image analysis software (OpenCV) to identify the materials and stores the results in a database as a text list. The analysis process uses object recognition algorithms and pre-trained models for material identification.

[1254] 3. Recipe suggestions and additional ingredients list

[1255] The server generates a recipe using a generative AI model (e.g., GPT-3) based on the generated ingredient list and the user's selected cooking type and number of days. The system also has a function to list additional ingredients that the user does not have in the recipe, allowing the user to visually check any missing ingredients.

[1256] 4. Food delivery orders

[1257] Users can check what ingredients they are missing on their device and order what they need via the food delivery service's API. The process includes real-time delivery time and cost estimates, and payment is processed using common electronic payment services.

[1258] 5. Food Photography Advice

[1259] Users can then upload photos of their completed dishes from their devices to the server, which uses image analysis software and generative AI models to generate feedback on the appearance and color of the dishes, improving the user experience.

[1260] Adding specific examples

[1261] Example prompts to input to the generative AI model

[1262] Prompt sentences from photo analysis results

[1263] The following ingredients were analyzed: chicken breast, broccoli, and tomato. Please suggest a diet recipe using these ingredients. The user wants to reduce sodium.

[1264] Food photography advice prompt

[1265] A user-uploaded food photo. Please provide feedback on the appearance and color of this dish.

[1266] This system allows users to efficiently utilize the ingredients they have on hand, create optimal recipes, and order missing ingredients for delivery all at once. Premium members can also receive customized recipes incorporating their specific preferences, as well as feedback on the appearance and quality of the finished dish, improving the user experience.

[1267] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1268] Step 1:

[1269] The user takes a photo of the material they have on hand using the device. As input, the user takes a photo of the material using the smartphone camera. This photo data is saved in the device application. It is desirable that the photo taken is high resolution.

[1270] Step 2:

[1271] The device uploads the captured photo data to the cloud server. The input is the photo data captured in step 1, and the output is the photo data uploaded to the cloud server. The upload process is securely performed via an internet connection.

[1272] Step 3:

[1273] The server receives the uploaded photos and analyzes the materials using image analysis software (OpenCV). The input is the uploaded photo data, and the output is a list of identified materials. The server saves the analysis results in a database in text format. The image analysis algorithm automatically identifies the materials in the photos.

[1274] Step 4:

[1275] The user selects the type of meal to be proposed (e.g., diet, regular menu, etc.) and the number of days on the terminal. The input is the information selected by the user on the terminal, and the output is the selection information being sent to the server. This includes operations on the user interface.

[1276] Step 5:

[1277] The server generates an optimal recipe using a generative AI model (e.g., GPT-3) based on the input ingredient list and user selection information. The input is the ingredient list and user selection information, and the output is a suggested recipe. The server inputs a prompt statement to the generative AI model to obtain the recipe.

[1278] Example prompt sentence:

[1279] The following ingredients were analyzed: chicken breast, broccoli, and tomato. Please suggest a diet recipe using these ingredients. The user wants to reduce sodium.

[1280] Step 6:

[1281] The server displays a list of additional ingredients that the user does not have based on the generated recipe. The input is the generated recipe, and the output is a list of missing ingredients. The list of missing ingredients is visually presented through a user interface.

[1282] Step 7:

[1283] The user selects the missing ingredients on the device and places an order via the food delivery service's API. The input is the list of missing ingredients and the user's selection. The output is order confirmation and delivery information. Payment processing is also included.

[1284] Step 8:

[1285] The user then takes a photo of the completed dish and uploads it to the cloud server. The input is the photo data of the completed dish, and the output is the photo data uploaded to the server.

[1286] Step 9:

[1287] The server analyzes the uploaded photos of the food using image analysis software and generative AI models to provide feedback on the appearance and color of the food. The input is the uploaded photo of the food, and the output is the feedback.

[1288] Example prompt sentence:

[1289] A user-uploaded food photo. Please provide feedback on the appearance and color of this dish.

[1290] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1291] This system allows users to take photos of ingredients they have on hand, analyzes the photos, and suggests optimal recipes. This system also provides feedback on the user's individual preferences and the cooking results. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized service.

[1292] System Overview

[1293] This system is realized through the cooperation of the user's "device," "server," and "emotion engine." The device used by the user can be a smartphone, tablet, or PC, and the server is a remote computer system that performs image analysis and recipe suggestions. The emotion engine is software or hardware that analyzes and reflects the user's emotions.

[1294] composition

[1295] 1. Photograph and upload the material

[1296] The user takes a photo of the ingredients to be used in the dish using the camera on the device.

[1297] The terminal provides an interface for uploading the captured photo data to the server.

[1298] 2. Materials Analysis

[1299] The server receives the uploaded photos and uses image analysis algorithms to identify the ingredients.

[1300] The server lists the identified materials and stores them in a database.

[1301] 3. Food suggestions

[1302] The user selects the type of food they would like suggested (diet, hearty menu, etc.) and the period (number of days) on the device.

[1303] The terminal transmits the user's selection information to the server.

[1304] The server searches the database for suitable recipes based on the ingredient list and user selection information.

[1305] The emotion engine analyzes the user's emotions and reflects them in recipe suggestions.

[1306] The server provides the suggested recipes to the user's terminal.

[1307] 4. Premium recipe suggestions

[1308] Premium members can also enter specific preferences (such as low salt or no oil).

[1309] The terminal transmits the desired conditions to the server.

[1310] The server generates a customized recipe incorporating the user's specific wishes and sends it to the user's device.

[1311] 5. Food Photography Advice

[1312] The user takes a photo of the finished dish with the device and uploads it to the server.

[1313] The server analyzes the uploaded photos and generates advice on the appearance and quality of the dish.

[1314] The emotion engine analyzes the user's emotions and reflects them in the advice.

[1315] The server provides the generated advice to the user's terminal.

[1316] Specific examples

[1317] For example, a user can take a photo of a day's worth of ingredients (chicken breast, broccoli, tomato, and sweet potato) and upload it to the server. After this, the user can request a diet menu for that day. The server analyzes the ingredients list, and the emotion engine analyzes the user's emotional state. If the server detects that the current mood is stressful, it will suggest recipes that have a stress-reducing effect. For example, it might suggest grilled chicken breast, steamed broccoli, tomato salad, and roasted sweet potato.

[1318] Furthermore, if a premium member user requests to reduce their salt intake, the server will provide low-salt recipes (e.g., low-salt grilled chicken breast, unsalted steamed broccoli, etc.). The emotion engine reflects the user's health concerns and prioritizes low-carb and low-calorie recipes.

[1319] After the dish is complete, the user uploads a photo of it to the server, which analyzes its appearance and color. The emotion engine then analyzes the user's emotional state and, if the user feels, for example, "I'm happy that it turned out well," provides advice on how to further improve the cooking technique (e.g., "Next time, it will look better if you balance the colors a little more").

[1320] By following these steps, users can efficiently use their own ingredients and create healthy and delicious meals. In addition, by analyzing and reflecting the user's emotions, they can receive more personalized service.

[1321] The processing flow will be explained below.

[1322] Specific processing flow of the program (system combining emotion engine)

[1323] Photographing and uploading materials

[1324] Step 1:

[1325] The user takes a photo of the ingredients to be used in cooking using the device's camera.

[1326] Step 2:

[1327] Temporarily saves photos taken by the device.

[1328] Step 3:

[1329] The user uses the interface of the terminal to perform an operation to upload a photo of the material to the server.

[1330] Step 4:

[1331] The device sends the photo data to the specified API endpoint.

[1332] Step 5:

[1333] The server receives and stores the uploaded photo data.

[1334] Materials Analysis

[1335] Step 6:

[1336] The server runs an image analysis algorithm on the received photo data to identify the material.

[1337] Step 7:

[1338] The server lists the identified materials and stores them in a database.

[1339] Step 8:

[1340] The server transmits the generated material list to the terminal and notifies the user.

[1341] Cooking suggestions

[1342] Step 9:

[1343] The user selects the type of food and number of days for which they would like suggestions on their device.

[1344] Step 10:

[1345] The terminal transmits the user's selection information to the server.

[1346] Step 11:

[1347] The server searches the database for an appropriate recipe based on the ingredients list and the user's selections.

[1348] Step 12:

[1349] The emotion engine analyzes the user's emotions and reflects them in recipe suggestions.

[1350] Step 13:

[1351] The server selects the most suitable recipes from the search results and the analysis results of the emotion engine, and generates a list of dishes.

[1352] Step 14:

[1353] The server sends the generated recipe list to the terminal and displays it to the user.

[1354] Premium recipe suggestions

[1355] Step 15:

[1356] A premium member user inputs their specific preferences (e.g., low salt, no oil) on the device.

[1357] Step 16:

[1358] The terminal transmits the desired conditions to the server.

[1359] Step 17:

[1360] The server searches the database for recipes that meet the desired conditions.

[1361] Step 18:

[1362] The emotion engine analyzes the user's emotions and reflects them in customized recipe suggestions.

[1363] Step 19:

[1364] The server selects recipes that meet the conditions and take into account the analysis results of the emotion engine, and generates a list of premium dishes.

[1365] Step 20:

[1366] The server transmits the generated premium dish list to the terminal and displays it to the user.

[1367] Food photography advice

[1368] Step 21:

[1369] The user takes a photo of the finished dish using the device.

[1370] Step 22:

[1371] The device sends the photo data to the specified API endpoint.

[1372] Step 23:

[1373] The server receives the uploaded photos and analyzes them.

[1374] Step 24:

[1375] The emotion engine analyzes the user's emotions and reflects them in the advice.

[1376] Step 25:

[1377] The server generates advice based on the results of the photo analysis and the emotion engine analysis.

[1378] Step 26:

[1379] The server sends the generated advice to the terminal and displays it to the user.

[1380] The above is the specific processing flow of the system that combines the emotion engine and the operations that are performed at each step.

[1381] Example 2

[1382] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1383] Conventional recipe suggestion systems lack the ability to provide personalized recipe suggestions based on the ingredients a user has available, and are unable to take into account the user's individual preferences and emotional state. Furthermore, there are no systems that provide feedback on the appearance or quality of a dish after it is completed.

[1384] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for uploading photos of ingredients from a terminal to the server, a means for the server to analyze the uploaded photos and generate an ingredient list, a means for the user to select a type of dish and a cooking period, a means for the server to suggest a recipe based on the ingredient list and the user's selection information, and a means for analyzing the user's emotional state using an emotion analysis means and reflecting the result in a recipe suggestion. This makes it possible to suggest personalized recipes based on the user's ingredients and provide feedback that takes the user's emotional state into consideration.

[1385] A "terminal" is a device that a user uses to take photos and upload data to a server, and includes a smartphone, tablet, PC, etc.

[1386] The "server" is a remote computer system that analyzes the photo data received from the terminal and generates ingredient lists and recipes.

[1387] "Image analysis algorithm" refers to the technology that recognizes and lists specific objects and materials from photographic data.

[1388] The "Ingredient List" is a list of ingredients obtained by analyzing photos uploaded from the terminal.

[1389] "User selection information" refers to information such as the type of food and the period selected by the user on the terminal.

[1390] "Emotion analysis means" refers to technology that analyzes the user's emotional state and reflects that in recipe suggestions and feedback.

[1391] A "recipe suggestion" is a recipe creation procedure generated by the server based on an ingredient list and user selections.

[1392] A "premium member" is a user who has the right to customize recipes by adding specific desired conditions.

[1393] "Feedback" is advice on the appearance and quality of the dish provided based on a photo of the finished dish.

[1394] System Overview

[1395] The present invention is a system that allows a user to take a photo of ingredients they have on hand, analyzes the photo, and suggests optimal recipes. Furthermore, it provides feedback on the user's individual preferences and the cooking results, and by using an emotion engine, it can provide a personalized service that responds to the user's emotions. This system mainly uses the following hardware and software:

[1396] Devices: Smartphones, tablets, computers

[1397] Server: A remote computer system (e.g., a cloud service)

[1398] Emotion Engine: Emotion Analysis Software

[1399] Program processing

[1400] The system's program includes a series of processes such as photographing and uploading ingredients, analyzing ingredients, suggesting dishes, suggesting premium recipes, and providing advice on food photography.

[1401] Photographing and uploading materials

[1402] The user takes a photo of the material at hand with the device's camera, and the device then uploads the photo to the server. This process includes an interface where the user can click a "Select Photo" button to select the photo they have taken.

[1403] Materials Analysis

[1404] The server receives the uploaded photo data and identifies the materials using image analysis algorithms (e.g., TensorFlow or OpenCV). The identified materials are listed and stored in a database.

[1405] Cooking suggestions

[1406] The user selects the type of food they want to be recommended (e.g., diet, hearty menu) and the period (number of days) on their device. The device sends the user's selection information to the server, which then suggests the optimal recipe based on the ingredients list and the user's selection information. In addition, the emotion engine analyzes the user's emotional state and reflects it in the recipe suggestions.

[1407] Premium recipe suggestions

[1408] Premium members can add specific preferences (e.g., less salt, no oil), and the device sends the preferences to the server, which then generates a customized recipe that takes the specific preferences into account.

[1409] Food photography advice

[1410] After cooking is complete, the user takes a photo of the finished dish and uploads it to the server. The server analyzes the photo and generates advice on the appearance and finish of the dish. The emotion engine analyzes the user's emotional state and reflects it in the advice.

[1411] Specific examples

[1412] For example, suppose a user has chicken breast, broccoli, tomato, and sweet potato. They take a photo of these ingredients and upload it to the server. When the user requests a daily diet menu, the server analyzes the ingredients. If the emotion engine determines that the user's emotional state is stressful, it will suggest recipes that have a stress-reducing effect. For example, grilled chicken breast, steamed broccoli, tomato salad, and roasted sweet potato may be suggested.

[1413] Additionally, if a premium member user requests reduced salt intake, the server will provide low-salt recipes (e.g., low-salt grilled chicken breast, unsalted steamed broccoli). The emotion engine reflects health-consciousness and prioritizes low-carb and low-calorie recipes. After the dish is complete, the user can upload a photo of it to the server, which will analyze its appearance and color and provide advice such as "Next time, you should try to balance the colors a little more" based on the user's emotion, such as "I'm happy that it looks good."

[1414] Prompt Sentence Examples

[1415] Here are some example prompts to input to a generative AI model:

[1416] Suggest a daily diet menu based on the ingredients the user has on hand (chicken breast, broccoli, tomato, sweet potato). Also, consider the user's current emotional state as stressful, and suggest recipes that will help reduce stress.

[1417] This allows users to efficiently use their ingredients and enjoy healthy cooking that is in line with their emotions.

[1418] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1419] Step 1: Photograph and upload your materials

[1420] Input: The user takes a photo of the ingredients at hand using the device's camera.

[1421] What it does: The user launches the camera application, clicks the "Take Photo" button to take a photo of the ingredient, and then selects the saved photo using the "Select Photo" function within the app.

[1422] Output: The selected photo data will be saved on your device.

[1423] Input: The device displays an interface for uploading the saved photo data to the server.

[1424] Specific behavior: The user clicks the "Upload Photo" button and the photo data is sent to the server.

[1425] Output: Uploaded photo data is sent to the server.

[1426] Step 2: Analyze the material

[1427] Input: The server receives the photo data sent from the device.

[1428] What happens: The server processes the HTTP request, decodes the photo data, and saves it.

[1429] Output: The saved photo data is passed to the image analysis module.

[1430] Input: The server uses image analysis algorithms (e.g. TensorFlow or OpenCV) to identify materials in the photo.

[1431] Specific operation: The server inputs the photo data into the image analysis model to identify and extract the material name.

[1432] Output: The identified materials are listed and stored in a database.

[1433] Step 3: Food suggestions

[1434] Input: The user selects the type of meal (e.g. diet, hearty) and the period (number of days) for which they would like suggestions on the device.

[1435] Specific behavior: The user makes a selection using a drop-down menu or radio buttons. After making a selection, the user clicks the "Submit" button.

[1436] Output: The selected data is sent from the device to the server.

[1437] Input: The server searches its database for a suitable recipe based on the ingredients list and user selections.

[1438] What it does: Executes an SQL query to extract the corresponding recipes from the database.

[1439] Output: The extracted recipe data is passed to the sentiment analysis module.

[1440] Input: The emotion engine analyzes the user's emotional state and reflects it in recipe suggestions.

[1441] What it does: The emotion engine analyzes the user's text input and past data to assess their current emotional state.

[1442] Output: The sentiment analysis results are added to the recipe data to generate the final recipe.

[1443] Input: The server sends the optimal recipe to the user's device.

[1444] Specific operation: Sends an HTTP response containing recipe data to the device.

[1445] Output: The user receives the suggested recipe on their device.

[1446] Step 4: Premium Recipe Proposal

[1447] Input: Premium members can additionally input specific preferences (e.g., low salt, no oil).

[1448] Specific actions: Enter your desired criteria using the text fields or checkboxes and click the "Submit" button.

[1449] Output: The desired condition data is sent from the terminal to the server.

[1450] Input: The server generates a customized recipe taking into account your desired conditions.

[1451] What it does: Adjusts SQL queries based on desired criteria to search and extract recipe data.

[1452] Output: The customized recipe data is sent to the user's device.

[1453] Step 5: Food Photography Tips

[1454] Input: The user takes a photo of the finished dish on their device and uploads it to the server.

[1455] Specific operations: Click the "Upload a photo of the finished dish" button, select a photo, and submit it.

[1456] Output: Photo data of the finished dish is sent to the server.

[1457] Input: The server receives the uploaded photo data.

[1458] Specific operation: Decodes and saves photo data.

[1459] Output: The saved photo data is passed to the image analysis module.

[1460] Input: The server analyzes the photo data and generates recommendations on the appearance and finish of the dish.

[1461] What it does: The image analysis model evaluates the photo and generates a feedback statement.

[1462] Output: The generated advice is passed to the sentiment analysis module.

[1463] Input: The emotion engine analyzes the user's emotional state and reflects it in the advice.

[1464] Specific operation: Integrate the sentiment analysis results into the feedback sentence.

[1465] Output: A feedback statement is generated for the user to receive.

[1466] (Application example 2)

[1467] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1468] Conventional recipe recommendation systems have been unable to flexibly suggest recipes based on the user's mood or physical condition, or when the user has a variety of cooking ingredients. There is a growing demand for recipe suggestions that take into account the passenger's emotions and current health status, especially in moving environments such as autonomous vehicles. However, no systems have been available to address this demand. This creates a need for personalized recipe suggestions that efficiently use ingredients carried by the user while also responding to the passenger's emotions and health status.

[1469] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading photos of ingredients from a terminal to the server, means for the server to analyze the uploaded photos and generate an ingredient list, means for the user to select the type of dish and the number of days, means for the server to suggest a recipe based on the ingredient list and the user's selection, means for a passenger to photograph and upload ingredients for the dish using a terminal in the autonomous vehicle, means for an emotion engine to analyze the passenger's emotions and reflect them in recipe suggestions, means for the server to suggest dishes that have effects such as stress reduction based on the passenger's current emotional state, means for generating recipes that incorporate specific requests for premium members, and means for the user to additionally input desired conditions (such as carbohydrate restriction or reduced salt). This makes it possible to efficiently use the ingredients that passengers have on hand while in an autonomous vehicle and to suggest personalized recipes that are optimal for the passenger's emotions and physical condition.

[1470] "Ingredients" refer to the ingredients and raw materials used to create a dish.

[1471] "Terminal" refers to electronic devices such as smartphones, tablets, and personal computers, which are devices that are directly operated by users.

[1472] A "server" is a computer system installed in the cloud or at a remote location, and is a device that analyzes and processes data.

[1473] "Upload" means sending data from the user's terminal to the server.

[1474] "Photo" refers to image data taken by a user using the camera function of the terminal.

[1475] "Image analysis" is a method by which the server analyzes photographic data to identify materials and other elements.

[1476] The "ingredient list" is a list of cooking ingredients identified through image analysis.

[1477] A "recipe" is a detailed description of the steps and ingredients needed to prepare a particular dish.

[1478] An "emotion engine" is a software or hardware system for analyzing a user's emotional state.

[1479] "Selection information" refers to conditions such as the type of food desired by the user and the number of days.

[1480] A "Premium Member" is a registered user who has access to additional services and customization features.

[1481] "Desired conditions" are requirements that the user specifically specifies when proposing a recipe (e.g., carbohydrate restriction, reduced salt, etc.).

[1482] An "autonomous vehicle" is a vehicle that operates automatically without a human driver.

[1483] "Passenger" means a user riding in an automated driving vehicle.

[1484] "Personalization" refers to customizing something to suit the preferences and needs of an individual user.

[1485] "Feedback" refers to the reactions and opinions of users regarding the services and suggestions provided.

[1486] The system of this invention allows users to photograph and upload ingredients they have, analyze them, and suggest optimal recipes. Furthermore, by combining it with an emotion engine, it is possible to suggest personalized recipes based on the user's emotional state and physical condition. Passengers in autonomous vehicles can receive cooking suggestions and advice tailored to their individual preferences using ingredients they carry with them while traveling.

[1487] Hardware and software used

[1488] Hardware

[1489] Smartphone (Android, iOS)

[1490] Information systems in autonomous vehicles

[1491] Cloud server (AWS, GCP, etc.)

[1492] software

[1493] Image analysis algorithms (TensorFlow, OpenCV)

[1494] Emotion engine (IBM Watson, Microsoft Azure Cognitive Services)

[1495] Recipe database (MySQL, PostgreSQL)

[1496] Mobile app development frameworks (Flutter, React Native)

[1497] System Operation

[1498] The user uses a smartphone to take a photo of the ingredients they are holding, and the photo data is uploaded to a cloud server via the autonomous vehicle's in-vehicle information system. The cloud server then uses image analysis algorithms to identify the ingredients in the photo and generate an ingredients list.

[1499] The user then selects the type of dish (e.g., diet menu, hearty menu, etc.) and the number of days (e.g., 1-day supply, 3-day supply), and sends this information from the device to the server. The server combines the selection information sent by the user with the ingredient list to search for the optimal recipe, and the emotion engine analyzes the user's emotional state and reflects it in the recipe suggestions.

[1500] For premium members, users can also enter specific preferences (e.g., low sugar, low salt), and this information is also sent to the server. The server then generates and provides customized recipes to users that take these preferences into account.

[1501] Once the dish is ready, the user takes a photo of the dish with their smartphone and uploads it to the server, which analyzes the photo and generates advice about the appearance and finish.The emotion engine analyzes the user's emotional state and reflects it in the advice.

[1502] Specific examples

[1503] For example, a passenger in an autonomous vehicle can take a photo of the chicken breast, tomatoes, and broccoli they purchased that day and upload it with their smartphone. At the same time, they can select a "diet menu" and input that they are currently feeling "stressed." The server analyzes the ingredient list, and the emotion engine suggests dishes that are effective in "reducing stress" (e.g., grilled chicken breast with a balanced salad).

[1504] Prompt sentence to input to the generative AI model:

[1505] I took photos of ingredients and uploaded them. I'd like a diet menu this time. I'm also feeling stressed right now. Please suggest recipes that take stress reduction into consideration.

[1506] The system allows passengers on the move to make the most of the ingredients they have on hand and easily create healthy and delicious meals tailored to their individual desires and emotional state.

[1507] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1508] Step 1:

[1509] The user uses a smartphone to take a photo of the ingredients to be used in cooking, launches the smartphone's camera app, and presses the capture button to generate photo data.

[1510] Input: smartphone, materials

[1511] Output: Photo data

[1512] Step 2:

[1513] The device uploads the captured photo data to the cloud server, and the device's application sends the photo data via the cloud service's API.

[1514] Input: Photo data

[1515] Output: Upload photo data to cloud server

[1516] Step 3:

[1517] The server receives the uploaded photos and uses image analysis algorithms to identify materials in the photos. The server performs image analysis using TensorFlow or OpenCV.

[1518] Input: Photo data

[1519] Output: Materials list

[1520] Step 4:

[1521] The server generates a material list and saves it in the database. The analyzed material information is saved in list format.

[1522] Input: Material data

[1523] Output: Material list saved in database

[1524] Step 5:

[1525] The user selects the type of food and the number of days they want to eat it using the device application. The user taps on the selection screen to select an item, and the selection information is generated.

[1526] Input: User selection information (food type, number of days)

[1527] Output: Selection information

[1528] Step 6:

[1529] The device sends the selection information to the cloud server. The device application calls an API to send the selection information to the server.

[1530] Input: Selection information

[1531] Output: Send to server

[1532] Step 7:

[1533] The server searches the database for the best recipe based on the ingredients list and selection information. It queries the database using an SQL query.

[1534] Input: Material list, selection information

[1535] Output: Recipe candidates

[1536] Step 8:

[1537] The emotion engine analyzes the user's emotional state and reflects it in recipe suggestions. The emotion engine obtains emotional data from the user's voice and facial expressions, analyzes it, and reflects it in recipe search results.

[1538] Input: Emotion data

[1539] Output: Personalized recipe suggestions

[1540] Step 9:

[1541] The server provides personalized recipes to the user's device, where the suggested recipes are sent to an application on the device and displayed on the screen.

[1542] Input: Personalized Recipe

[1543] Output: Recipe display

[1544] Step 10:

[1545] For premium members, users can enter additional specific conditions they desire (e.g., carbohydrate restriction, reduced salt intake, etc.). Desired condition data is generated when users enter additional desired conditions into the terminal.

[1546] Input: Desired conditions

[1547] Output: Desired condition data

[1548] Step 11:

[1549] The device sends the desired conditions data to the server. The device application calls the API to send the desired conditions data to the server.

[1550] Input: Desired conditions data

[1551] Output: Send to server

[1552] Step 12:

[1553] The server generates a customized recipe based on the user's desired conditions and provides it to the user's device. The server then searches the database again, selects a recipe that meets the desired conditions, and sends it to the device.

[1554] Input: Desired conditions, material list

[1555] Output: Customized recipe

[1556] Step 13:

[1557] Once the user has completed the cooking, they take a photo of it with their smartphone and upload it to the server. They then use the smartphone's camera app to take another photo and upload it to the cloud server.

[1558] Input: Food photo

[1559] Output: Upload photos to cloud server

[1560] Step 14:

[1561] The server analyzes the uploaded photos of the finished dish, and the emotion engine analyzes the user's emotional state and reflects it in the advice. The server uses an image analysis algorithm, and the emotion engine analyzes the user's emotional data.

[1562] Input: Photos of finished dishes, user emotion data

[1563] Output: Advice

[1564] Step 15:

[1565] The server provides the generated advice to the user's terminal, where it is sent to an application on the terminal and displayed to the user.

[1566] Input: Advice

[1567] Output: Advice display

[1568] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1569] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1570] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1571] [Fourth embodiment]

[1572] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1573] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1574] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1575] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1576] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1577] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1578] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1579] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1580] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1581] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1582] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1583] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1584] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1585] The system allows users to take photos of ingredients they have on hand, analyzes the photos, and suggests optimal recipes. The system also provides feedback on the user's individual preferences and the cooking results.

[1586] System Overview

[1587] This system is realized through the cooperation of the user's "device" and a "server." The device used by the user can be a smartphone, tablet, or PC, while the server is a remote computer system that performs image analysis and recipe suggestions.

[1588] composition

[1589] 1. Photograph and upload the material

[1590] The user takes a photo of the ingredients to be used in the dish using the camera on the device.

[1591] The terminal provides an interface for uploading the captured photo data to the server.

[1592] 2. Materials Analysis

[1593] The server receives the uploaded photos and uses image analysis algorithms to identify the ingredients.

[1594] The server lists the identified materials and stores them in a database.

[1595] 3. Food suggestions

[1596] The user selects the type of food they would like suggested (diet, hearty menu, etc.) and the period (number of days) on the device.

[1597] The terminal transmits the user's selection information to the server.

[1598] The server retrieves suitable recipes from a database based on the ingredient list and user selections.

[1599] The server provides the suggested recipes to the user's terminal.

[1600] 4. Premium recipe suggestions

[1601] Premium members can also enter specific preferences (such as low salt or no oil).

[1602] The server generates a customized recipe incorporating the user's specific wishes and sends it to the user's device.

[1603] 5. Food Photography Advice

[1604] The user takes a photo of the finished dish with the device and uploads it to the server.

[1605] The server analyzes the uploaded photos and generates advice on the appearance and quality of the dish.

[1606] The server provides the generated advice to the user's terminal.

[1607] Specific examples

[1608] For example, a user can take a photo of a day's worth of ingredients (chicken breast, broccoli, tomato, sweet potato) and upload it to the server. After this, the user can request a diet menu for the day. The server analyzes the ingredient list and suggests appropriate recipes for dieters, such as grilled chicken breast, steamed broccoli, tomato salad, and roasted sweet potato.

[1609] Additionally, if premium members add a preference for reduced salt intake, the server will provide recipes that are suitable for reduced salt intake (e.g., low-sodium grilled chicken breast, unsalted steamed broccoli, etc.).

[1610] Once the dish is complete, the user can upload a photo of it to the server, which can then analyze its appearance and color and provide advice on how to improve it, such as using a marinade to keep the food moist.

[1611] By following these steps, users can efficiently use their ingredients to create healthy and delicious meals.

[1612] The processing flow will be explained below.

[1613] Specific processing flow of the program

[1614] Photographing and uploading materials

[1615] Step 1:

[1616] The user takes a photo of the ingredients to be used in cooking using the device's camera.

[1617] Step 2:

[1618] Temporarily saves photos taken by the device.

[1619] Step 3:

[1620] The user uses the interface of the terminal to perform an operation to upload a photo of the material to the server.

[1621] Step 4:

[1622] The device sends the photo data to the specified API endpoint.

[1623] Step 5:

[1624] The server receives and stores the uploaded photo data.

[1625] Materials Analysis

[1626] Step 6:

[1627] The server runs an image analysis algorithm on the received photo data to identify the material.

[1628] Step 7:

[1629] The server lists the identified materials and stores them in a database.

[1630] Step 8:

[1631] The server transmits the generated material list to the terminal and notifies the user.

[1632] Cooking suggestions

[1633] Step 9:

[1634] The user selects the type of food and number of days for which they would like suggestions on their device.

[1635] Step 10:

[1636] The terminal transmits the user's selection information to the server.

[1637] Step 11:

[1638] The server searches the database for an appropriate recipe based on the ingredients list and the user's selections.

[1639] Step 12:

[1640] The server selects the most suitable recipe from the search results and generates a list of dishes.

[1641] Step 13:

[1642] The server sends the generated recipe list to the terminal and displays it to the user.

[1643] Premium recipe suggestions

[1644] Step 14:

[1645] A premium member user inputs their specific preferences (e.g., low salt, no oil) on the device.

[1646] Step 15:

[1647] The terminal transmits the desired conditions to the server.

[1648] Step 16:

[1649] The server searches the database for recipes that meet the desired conditions.

[1650] Step 17:

[1651] The server selects recipes that meet the conditions and generates a list of premium dishes.

[1652] Step 18:

[1653] The server transmits the generated premium dish list to the terminal and displays it to the user.

[1654] Food photography advice

[1655] Step 19:

[1656] The user takes a photo of the finished dish using the device.

[1657] Step 20:

[1658] The device sends the photo data to the specified API endpoint.

[1659] Step 21:

[1660] The server receives the uploaded photos and analyzes them.

[1661] Step 22:

[1662] The server generates advice based on the results of analyzing the photo.

[1663] Step 23:

[1664] The server sends the generated advice to the terminal and displays it to the user.

[1665] The above is the specific processing flow of the system and the operations performed at each step.

[1666] Example 1

[1667] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1668] In today's busy lifestyles, users are required to maximize the use of ingredients on hand to create efficient and healthy meals. However, the process from selecting ingredients to searching for recipes and receiving feedback after cooking is cumbersome and time-consuming. It is also difficult to find recipes that meet specific nutritional restrictions or preferences. Therefore, there is a need for a system that can easily suggest healthy and varied meals using ingredients on hand, and also evaluate and improve the quality of the meals.

[1669] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1670] In this invention, the server includes means for uploading photos of ingredients from a terminal to the data processing device, means for the data processing device to analyze the uploaded photos and generate an ingredient list, means for a user to select a type of dish and a cooking period, means for the data processing device to suggest a food preparation method based on the ingredient list and the user's selection, means for generating a customized food preparation method using a generative AI model based on the user's specific preferences, and means for analyzing the uploaded photos of the finished food and generating advice, thereby enabling users to efficiently use ingredients on hand to create healthy and diverse dishes and receive specific feedback on the results.

[1671] "Photo of ingredients" is image data of ingredients used in cooking that the user has photographed with the camera on the terminal.

[1672] A "terminal" is an electronic device used by a user, such as a smartphone, tablet, or PC.

[1673] A "data processing device" is a remote computer system such as a server or a cloud system.

[1674] The "material list" is data that lists the names of materials analyzed by the data processing device.

[1675] A "user" is an individual who uses this system to input ingredients and receive cooking suggestions.

[1676] "Type of food" is a specific food category selected by the user, such as diet menu or hearty menu.

[1677] The "period" is the number of days for which the user wishes to receive suggestions for selected dishes.

[1678] "Food preparation method" is a recipe suggested based on ingredients and conditions entered by the user.

[1679] A "generative AI model" is an artificial intelligence model that generates new data or text based on given information.

[1680] A "customized food recipe" is a recipe generated by a generative AI model that incorporates a user's specific preferences (e.g., low salt, no oil).

[1681] "Advice" is feedback that indicates improvements and suggestions for the appearance and finish of the dish based on the analysis results.

[1682] A "Premium Member" is a user who has special access rights to use certain additional features and services.

[1683] The present invention is a system that takes a photo of the ingredients a user has on hand, analyzes the photo, and suggests optimal recipes. The system also provides feedback on the user's individual preferences and the cooking results.

[1684] This system is realized through the cooperation of a server and a device used by the user. The device used by the user can be a smartphone, tablet, or PC, and the server is a remote computer system that performs image analysis and recipe suggestions.

[1685] Photographing and uploading materials

[1686] The user takes a photo of the ingredients to be used in the dish using the camera of the device, for example, using a camera app on a smartphone.

[1687] The device uploads the captured photo data to the server, which is sent over the Internet using an HTTP POST request.

[1688] Materials Analysis

[1689] The server receives the uploaded photos and identifies the materials using image analysis algorithms (e.g., TensorFlow or OpenCV).

[1690] The server lists the identified materials and stores them in a database (e.g., MySQL or PostgreSQL).

[1691] Cooking suggestions

[1692] The user selects the type of meal they want to be suggested (diet, hearty, etc.) and the period (number of days) on the device, for example, using a drop-down menu in the UI.

[1693] The terminal transmits the user's selection information to the server.

[1694] The server searches the database for suitable recipes based on the ingredients list and user selections, extracting data using SQL queries.

[1695] The server sends the search results in JSON format to the user's device, which receives them and displays them on the application screen.

[1696] Premium recipe suggestions

[1697] Premium users input their specific preferences, such as low salt or no oil, into the terminal application, for example using text fields or check boxes.

[1698] The terminal transmits this input information to the server.

[1699] The server uses a generative AI model (e.g., GPT-3) to generate a customized recipe based on your specific preferences.

[1700] The server transmits the generated customized recipe to the user's terminal and displays it on the terminal application.

[1701] Food photography advice

[1702] Users take a photo of the finished dish with their smartphone camera and use the application to upload it to the server, where the photo is sent using an HTTP request.

[1703] The server analyzes the received photos and performs an evaluation using image analysis algorithms.

[1704] The server generates advice based on the evaluation results, including suggestions such as "marinate to keep the food moist" or "add parsley for color."

[1705] The server sends the generated advice as a text message to the user's terminal, which displays it.

[1706] By following these steps, users can efficiently use their ingredients to create healthy and delicious meals.

[1707] Examples of prompt statements

[1708] "Suggest a day's worth of meals suitable for today's diet"

[1709] "I'd like a low-sodium menu, so please tell me the recipe."

[1710] "Can you give me some advice on how to improve the appearance of this dish?"

[1711] This allows users to easily obtain recipes and advice that meet their needs.

[1712] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1713] Step 1:

[1714] The user takes a photo of the ingredients to be used in cooking using the camera app on their smartphone. The photo of the ingredients is obtained as input. The device displays the photo data on a preview screen within the app, and when the user is satisfied, they tap the "Upload" button. The photo data is sent to the server as output.

[1715] Step 2:

[1716] The device sends the photo data taken by the user to the server via the Internet. Specifically, it uses an HTTP POST request to send image data to the server. It receives the photo data as input and sends it to the server as output.

[1717] Step 3:

[1718] The server stores the photo data received from the device and applies image analysis algorithms (e.g., TensorFlow or OpenCV) to identify materials. The photo data is taken as input and a list of materials is generated through analysis. This list of materials is then generated as output and stored in a database.

[1719] Step 4:

[1720] The user selects the type of food they want to be recommended (for example, diet menu or hearty menu) and the period (number of days) on the terminal application. The user's selection information is obtained as input. The selection information is sent to the server as output.

[1721] Step 5:

[1722] The terminal transmits the user's selection information to the server. Specifically, the terminal transmits the selection information using an HTTP request. The terminal receives the user's selection information as input and transmits it to the server.

[1723] Step 6:

[1724] The server uses the ingredient list and user selection information to search the database for suitable recipes. Specifically, it uses SQL queries to extract data. The ingredient list and user selection information are received as input, and suitable recipe information is generated by performing a database search. This recipe information is generated as output.

[1725] Step 7:

[1726] The server sends the search results in JSON format to the user's device. It receives the generated recipe information as input and sends it to the device as JSON format data as output.

[1727] Step 8:

[1728] The device receives the recipe information and displays it on the application screen. It receives recipe information in JSON format as input and displays it on the application screen as output.

[1729] Step 9:

[1730] Premium members input their specific preferences, such as low salt or no oil, into the terminal application. The specific preferences are obtained as input. The preferences are sent to the server as output.

[1731] Step 10:

[1732] The terminal sends this input information to the server. Specifically, it sends the desired information using an HTTP request. It receives specific desired information as input and sends it to the server.

[1733] Step 11:

[1734] The server generates a customized recipe using a generative AI model (e.g., GPT-3) based on the user's specific preferences. The input is the user's preferences, and the output is the customized recipe information.

[1735] Step 12:

[1736] The server sends the generated customized recipe to the user's terminal and displays it in the terminal application, which receives the customized recipe information as input and sends it to the terminal as output.

[1737] Step 13:

[1738] The user takes a photo of the finished dish with their smartphone camera and uploads it to the server using the application. Similarly, the photo is sent using an HTTP request, which takes the photo of the finished dish as input and sends it to the server as output.

[1739] Step 14:

[1740] The server analyzes the received photos and uses image analysis algorithms to generate a rating, taking a photo of the finished dish as input and generating a rating result as output.

[1741] Step 15:

[1742] The server generates advice based on the evaluation results and suggests useful methods and improvements to the user.The evaluation results are taken as input and advice is generated as output.

[1743] Step 16:

[1744] The server sends this advice as a text message to the user's terminal, which displays it.The advice generated is taken as input and sent to the terminal as output.

[1745] (Application example 1)

[1746] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1747] Conventional ingredient analysis and recipe suggestion systems lacked a simple way for users to order additional ingredients when they needed more than they could use with the ingredients they already had on hand. They also struggled to provide customized recipes that accommodate users' dietary restrictions or specific preferences, leaving the user experience open to improvement. Furthermore, they lacked support for improving the quality of their cooking, as they had limited means of receiving feedback on the appearance and finish of the finished dish.

[1748] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1749] In this invention, the server includes a means for uploading photos of ingredients from a terminal to the server, a means for the server to analyze the uploaded photos and generate an ingredient list, a means for the user to select a type of dish and the number of days for cooking, a means for the server to suggest a recipe based on the ingredient list and the user's selection, a means for displaying a list of additional ingredients for the suggested recipe, and a means for ordering missing ingredients via a food delivery service. This allows users to efficiently utilize the ingredients they have on hand and order optimal recipes and missing ingredients for delivery all at once. Premium members can also receive customized recipes incorporating their specific preferences and feedback on the appearance and quality of the finished dish, thereby improving the user experience.

[1750] The "means for uploading a photo of an ingredient from a terminal to a server" is a function that allows a user to take a photo of an ingredient using his or her own terminal and transmit the data to a remote server.

[1751] "Means for the server to analyze uploaded photos and generate an ingredient list" refers to a function that uses an image analysis algorithm to analyze uploaded photo data on the server side, identify the ingredients in the photos, and create a list.

[1752] "Means for users to select the type of dish and number of days" refers to an interface that allows users to input or select the type of dish they want to make (e.g., diet, nutritional balance, etc.) and the period (number of days) for which they want to receive recipe suggestions.

[1753] "Means for the server to suggest recipes based on the ingredient list and user selection" refers to a function in which the server uses a database or generative AI model to generate and suggest optimal cooking recipes based on the generated ingredient list and user selection.

[1754] "Means for displaying a list of additional ingredients for a proposed recipe" refers to an interface that displays a list of additional ingredients that the user does not currently have based on the proposed recipe, allowing the user to visually confirm them.

[1755] The "means for ordering missing ingredients via a food delivery service" refers to a system for selecting the missing ingredients from the list, ordering those ingredients via the corresponding food delivery service, and arranging for delivery.

[1756] The embodiment of this invention is a system that proposes optimal recipes based on ingredients on hand and allows users to order additional ingredients via a food delivery service. This system includes the following main hardware and software components:

[1757] Hardware and software used

[1758] Smartphone (user device)

[1759] Cloud server (image analysis and recipe suggestions)

[1760] Image analysis software (OpenCV)

[1761] Machine learning models (TensorFlow and PyTorch)

[1762] Database (PostgreSQL, MongoDB)

[1763] Food delivery API (e.g. Uber Eats, DoorDash)

[1764] Generative AI models (e.g., GPT-3)

[1765] Specific explanation of the system

[1766] 1. Take and upload a photo of the material

[1767] Users take photos of materials they have on hand using their smartphones. The device then uploads the captured photo data to a cloud server. Users can use features such as laser focus assistance and brightness adjustment filters to improve analysis accuracy.

[1768] 2. Materials Analysis

[1769] The cloud server analyzes the uploaded photo data using image analysis software (OpenCV) to identify the materials and stores the results in a database as a text list. The analysis process uses object recognition algorithms and pre-trained models for material identification.

[1770] 3. Recipe suggestions and additional ingredients list

[1771] The server generates a recipe using a generative AI model (e.g., GPT-3) based on the generated ingredient list and the user's selected cooking type and number of days. The system also has a function to list additional ingredients that the user does not have in the recipe, allowing the user to visually check any missing ingredients.

[1772] 4. Food delivery orders

[1773] Users can check what ingredients they are missing on their device and order what they need via the food delivery service's API. The process includes real-time delivery time and cost estimates, and payment is processed using common electronic payment services.

[1774] 5. Food Photography Advice

[1775] Users can then upload photos of their completed dishes from their devices to the server, which uses image analysis software and generative AI models to generate feedback on the appearance and color of the dishes, improving the user experience.

[1776] Adding specific examples

[1777] Example prompts to input to the generative AI model

[1778] Prompt sentences from photo analysis results

[1779] The following ingredients were analyzed: chicken breast, broccoli, and tomato. Please suggest a diet recipe using these ingredients. The user wants to reduce sodium.

[1780] Food photography advice prompt

[1781] A user-uploaded food photo. Please provide feedback on the appearance and color of this dish.

[1782] This system allows users to efficiently utilize the ingredients they have on hand, create optimal recipes, and order missing ingredients for delivery all at once. Premium members can also receive customized recipes incorporating their specific preferences, as well as feedback on the appearance and quality of the finished dish, improving the user experience.

[1783] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1784] Step 1:

[1785] The user takes a photo of the material they have on hand using the device. As input, the user takes a photo of the material using the smartphone camera. This photo data is saved in the device application. It is desirable that the photo taken is high resolution.

[1786] Step 2:

[1787] The device uploads the captured photo data to the cloud server. The input is the photo data captured in step 1, and the output is the photo data uploaded to the cloud server. The upload process is securely performed via an internet connection.

[1788] Step 3:

[1789] The server receives the uploaded photos and analyzes the materials using image analysis software (OpenCV). The input is the uploaded photo data, and the output is a list of identified materials. The server saves the analysis results in a database in text format. The image analysis algorithm automatically identifies the materials in the photos.

[1790] Step 4:

[1791] The user selects the type of meal to be proposed (e.g., diet, regular menu, etc.) and the number of days on the terminal. The input is the information selected by the user on the terminal, and the output is the selection information being sent to the server. This includes operations on the user interface.

[1792] Step 5:

[1793] The server generates an optimal recipe using a generative AI model (e.g., GPT-3) based on the input ingredient list and user selection information. The input is the ingredient list and user selection information, and the output is a suggested recipe. The server inputs a prompt statement to the generative AI model to obtain the recipe.

[1794] Example prompt sentence:

[1795] The following ingredients were analyzed: chicken breast, broccoli, and tomato. Please suggest a diet recipe using these ingredients. The user wants to reduce sodium.

[1796] Step 6:

[1797] The server displays a list of additional ingredients that the user does not have based on the generated recipe. The input is the generated recipe, and the output is a list of missing ingredients. The list of missing ingredients is visually presented through a user interface.

[1798] Step 7:

[1799] The user selects the missing ingredients on the device and places an order via the food delivery service's API. The input is the list of missing ingredients and the user's selection. The output is order confirmation and delivery information. Payment processing is also included.

[1800] Step 8:

[1801] The user then takes a photo of the completed dish and uploads it to the cloud server. The input is the photo data of the completed dish, and the output is the photo data uploaded to the server.

[1802] Step 9:

[1803] The server analyzes the uploaded photos of the food using image analysis software and generative AI models to provide feedback on the appearance and color of the food. The input is the uploaded photo of the food, and the output is the feedback.

[1804] Example prompt sentence:

[1805] A user-uploaded food photo. Please provide feedback on the appearance and color of this dish.

[1806] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1807] This system allows users to take photos of ingredients they have on hand, analyzes the photos, and suggests optimal recipes. This system also provides feedback on the user's individual preferences and the cooking results. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized service.

[1808] System Overview

[1809] This system is realized through the cooperation of the user's "device," "server," and "emotion engine." The device used by the user can be a smartphone, tablet, or PC, and the server is a remote computer system that performs image analysis and recipe suggestions. The emotion engine is software or hardware that analyzes and reflects the user's emotions.

[1810] composition

[1811] 1. Photograph and upload the material

[1812] The user takes a photo of the ingredients to be used in the dish using the camera on the device.

[1813] The terminal provides an interface for uploading the captured photo data to the server.

[1814] 2. Materials Analysis

[1815] The server receives the uploaded photos and uses image analysis algorithms to identify the ingredients.

[1816] The server lists the identified materials and stores them in a database.

[1817] 3. Food suggestions

[1818] The user selects the type of food they would like suggested (diet, hearty menu, etc.) and the period (number of days) on the device.

[1819] The terminal transmits the user's selection information to the server.

[1820] The server searches the database for suitable recipes based on the ingredient list and user selection information.

[1821] The emotion engine analyzes the user's emotions and reflects them in recipe suggestions.

[1822] The server provides the suggested recipes to the user's terminal.

[1823] 4. Premium recipe suggestions

[1824] Premium members can also enter specific preferences (such as low salt or no oil).

[1825] The terminal transmits the desired conditions to the server.

[1826] The server generates a customized recipe incorporating the user's specific wishes and sends it to the user's device.

[1827] 5. Food Photography Advice

[1828] The user takes a photo of the finished dish with the device and uploads it to the server.

[1829] The server analyzes the uploaded photos and generates advice on the appearance and quality of the dish.

[1830] The emotion engine analyzes the user's emotions and reflects them in the advice.

[1831] The server provides the generated advice to the user's terminal.

[1832] Specific examples

[1833] For example, a user can take a photo of a day's worth of ingredients (chicken breast, broccoli, tomato, and sweet potato) and upload it to the server. After this, the user can request a diet menu for that day. The server analyzes the ingredients list, and the emotion engine analyzes the user's emotional state. If the server detects that the current mood is stressful, it will suggest recipes that have a stress-reducing effect. For example, it might suggest grilled chicken breast, steamed broccoli, tomato salad, and roasted sweet potato.

[1834] Furthermore, if a premium member user requests to reduce their salt intake, the server will provide low-salt recipes (e.g., low-salt grilled chicken breast, unsalted steamed broccoli, etc.). The emotion engine reflects the user's health concerns and prioritizes low-carb and low-calorie recipes.

[1835] After the dish is complete, the user uploads a photo of it to the server, which analyzes its appearance and color. The emotion engine then analyzes the user's emotional state and, if the user feels, for example, "I'm happy that it turned out well," provides advice on how to further improve the cooking technique (e.g., "Next time, it will look better if you balance the colors a little more").

[1836] By following these steps, users can efficiently use their own ingredients and create healthy and delicious meals. In addition, by analyzing and reflecting the user's emotions, they can receive more personalized service.

[1837] The processing flow will be explained below.

[1838] Specific processing flow of the program (system combining emotion engine)

[1839] Photographing and uploading materials

[1840] Step 1:

[1841] The user takes a photo of the ingredients to be used in cooking using the device's camera.

[1842] Step 2:

[1843] Temporarily saves photos taken by the device.

[1844] Step 3:

[1845] The user uses the interface of the terminal to perform an operation to upload a photo of the material to the server.

[1846] Step 4:

[1847] The device sends the photo data to the specified API endpoint.

[1848] Step 5:

[1849] The server receives and stores the uploaded photo data.

[1850] Materials Analysis

[1851] Step 6:

[1852] The server runs an image analysis algorithm on the received photo data to identify the material.

[1853] Step 7:

[1854] The server lists the identified materials and stores them in a database.

[1855] Step 8:

[1856] The server transmits the generated material list to the terminal and notifies the user.

[1857] Cooking suggestions

[1858] Step 9:

[1859] The user selects the type of food and number of days for which they would like suggestions on their device.

[1860] Step 10:

[1861] The terminal transmits the user's selection information to the server.

[1862] Step 11:

[1863] The server searches the database for an appropriate recipe based on the ingredients list and the user's selections.

[1864] Step 12:

[1865] The emotion engine analyzes the user's emotions and reflects them in recipe suggestions.

[1866] Step 13:

[1867] The server selects the most suitable recipes from the search results and the analysis results of the emotion engine, and generates a list of dishes.

[1868] Step 14:

[1869] The server sends the generated recipe list to the terminal and displays it to the user.

[1870] Premium recipe suggestions

[1871] Step 15:

[1872] A premium member user inputs their specific preferences (e.g., low salt, no oil) on the device.

[1873] Step 16:

[1874] The terminal transmits the desired conditions to the server.

[1875] Step 17:

[1876] The server searches the database for recipes that meet the desired conditions.

[1877] Step 18:

[1878] The emotion engine analyzes the user's emotions and reflects them in customized recipe suggestions.

[1879] Step 19:

[1880] The server selects recipes that meet the conditions and take into account the analysis results of the emotion engine, and generates a list of premium dishes.

[1881] Step 20:

[1882] The server transmits the generated premium dish list to the terminal and displays it to the user.

[1883] Food photography advice

[1884] Step 21:

[1885] The user takes a photo of the finished dish using the device.

[1886] Step 22:

[1887] The device sends the photo data to the specified API endpoint.

[1888] Step 23:

[1889] The server receives the uploaded photos and analyzes them.

[1890] Step 24:

[1891] The emotion engine analyzes the user's emotions and reflects them in the advice.

[1892] Step 25:

[1893] The server generates advice based on the results of the photo analysis and the emotion engine analysis.

[1894] Step 26:

[1895] The server sends the generated advice to the terminal and displays it to the user.

[1896] The above is the specific processing flow of the system that combines the emotion engine and the operations that are performed at each step.

[1897] Example 2

[1898] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1899] Conventional recipe suggestion systems lack the ability to provide personalized recipe suggestions based on the ingredients a user has available, and are unable to take into account the user's individual preferences and emotional state. Furthermore, there are no systems that provide feedback on the appearance or quality of a dish after it is completed.

[1900] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for uploading photos of ingredients from a terminal to the server, a means for the server to analyze the uploaded photos and generate an ingredient list, a means for the user to select a type of dish and a cooking period, a means for the server to suggest a recipe based on the ingredient list and the user's selection information, and a means for analyzing the user's emotional state using an emotion analysis means and reflecting the result in a recipe suggestion. This makes it possible to suggest personalized recipes based on the user's ingredients and provide feedback that takes the user's emotional state into consideration.

[1901] A "terminal" is a device that a user uses to take photos and upload data to a server, and includes a smartphone, tablet, PC, etc.

[1902] The "server" is a remote computer system that analyzes the photo data received from the terminal and generates ingredient lists and recipes.

[1903] "Image analysis algorithm" refers to the technology that recognizes and lists specific objects and materials from photographic data.

[1904] The "Ingredient List" is a list of ingredients obtained by analyzing photos uploaded from the terminal.

[1905] "User selection information" refers to information such as the type of food and the period selected by the user on the terminal.

[1906] "Emotion analysis means" refers to technology that analyzes the user's emotional state and reflects that in recipe suggestions and feedback.

[1907] A "recipe suggestion" is a recipe creation procedure generated by the server based on an ingredient list and user selections.

[1908] A "premium member" is a user who has the right to customize recipes by adding specific desired conditions.

[1909] "Feedback" is advice on the appearance and quality of the dish provided based on a photo of the finished dish.

[1910] System Overview

[1911] The present invention is a system that allows a user to take a photo of ingredients they have on hand, analyzes the photo, and suggests optimal recipes. Furthermore, it provides feedback on the user's individual preferences and the cooking results, and by using an emotion engine, it can provide a personalized service that responds to the user's emotions. This system mainly uses the following hardware and software:

[1912] Devices: Smartphones, tablets, computers

[1913] Server: A remote computer system (e.g., a cloud service)

[1914] Emotion Engine: Emotion Analysis Software

[1915] Program processing

[1916] The system's program includes a series of processes such as photographing and uploading ingredients, analyzing ingredients, suggesting dishes, suggesting premium recipes, and providing advice on food photography.

[1917] Photographing and uploading materials

[1918] The user takes a photo of the material at hand with the device's camera, and the device then uploads the photo to the server. This process includes an interface where the user can click a "Select Photo" button to select the photo they have taken.

[1919] Materials Analysis

[1920] The server receives the uploaded photo data and identifies the materials using image analysis algorithms (e.g., TensorFlow or OpenCV). The identified materials are listed and stored in a database.

[1921] Cooking suggestions

[1922] The user selects the type of food they want to be recommended (e.g., diet, hearty menu) and the period (number of days) on their device. The device sends the user's selection information to the server, which then suggests the optimal recipe based on the ingredients list and the user's selection information. In addition, the emotion engine analyzes the user's emotional state and reflects it in the recipe suggestions.

[1923] Premium recipe suggestions

[1924] Premium members can add specific preferences (e.g., less salt, no oil), and the device sends the preferences to the server, which then generates a customized recipe that takes the specific preferences into account.

[1925] Food photography advice

[1926] After cooking is complete, the user takes a photo of the finished dish and uploads it to the server. The server analyzes the photo and generates advice on the appearance and finish of the dish. The emotion engine analyzes the user's emotional state and reflects it in the advice.

[1927] Specific examples

[1928] For example, suppose a user has chicken breast, broccoli, tomato, and sweet potato. They take a photo of these ingredients and upload it to the server. When the user requests a daily diet menu, the server analyzes the ingredients. If the emotion engine determines that the user's emotional state is stressful, it will suggest recipes that have a stress-reducing effect. For example, grilled chicken breast, steamed broccoli, tomato salad, and roasted sweet potato may be suggested.

[1929] Additionally, if a premium member user requests reduced salt intake, the server will provide low-salt recipes (e.g., low-salt grilled chicken breast, unsalted steamed broccoli). The emotion engine reflects health-consciousness and prioritizes low-carb and low-calorie recipes. After the dish is complete, the user can upload a photo of it to the server, which will analyze its appearance and color and provide advice such as "Next time, you should try to balance the colors a little more" based on the user's emotion, such as "I'm happy that it looks good."

[1930] Prompt Sentence Examples

[1931] Here are some example prompts to input to a generative AI model:

[1932] Suggest a daily diet menu based on the ingredients the user has on hand (chicken breast, broccoli, tomato, sweet potato). Also, consider the user's current emotional state as stressful, and suggest recipes that will help reduce stress.

[1933] This allows users to efficiently use their ingredients and enjoy healthy cooking that is in line with their emotions.

[1934] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1935] Step 1: Photograph and upload your materials

[1936] Input: The user takes a photo of the ingredients at hand using the device's camera.

[1937] What it does: The user launches the camera application, clicks the "Take Photo" button to take a photo of the ingredient, and then selects the saved photo using the "Select Photo" function within the app.

[1938] Output: The selected photo data will be saved on your device.

[1939] Input: The device displays an interface for uploading the saved photo data to the server.

[1940] Specific behavior: The user clicks the "Upload Photo" button and the photo data is sent to the server.

[1941] Output: Uploaded photo data is sent to the server.

[1942] Step 2: Analyze the material

[1943] Input: The server receives the photo data sent from the device.

[1944] What happens: The server processes the HTTP request, decodes the photo data, and saves it.

[1945] Output: The saved photo data is passed to the image analysis module.

[1946] Input: The server uses image analysis algorithms (e.g. TensorFlow or OpenCV) to identify materials in the photo.

[1947] Specific operation: The server inputs the photo data into the image analysis model to identify and extract the material name.

[1948] Output: The identified materials are listed and stored in a database.

[1949] Step 3: Food suggestions

[1950] Input: The user selects the type of meal (e.g. diet, hearty) and the period (number of days) for which they would like suggestions on the device.

[1951] Specific behavior: The user makes a selection using a drop-down menu or radio buttons. After making a selection, the user clicks the "Submit" button.

[1952] Output: The selected data is sent from the device to the server.

[1953] Input: The server searches its database for a suitable recipe based on the ingredients list and user selections.

[1954] What it does: Executes an SQL query to extract the corresponding recipes from the database.

[1955] Output: The extracted recipe data is passed to the sentiment analysis module.

[1956] Input: The emotion engine analyzes the user's emotional state and reflects it in recipe suggestions.

[1957] What it does: The emotion engine analyzes the user's text input and past data to assess their current emotional state.

[1958] Output: The sentiment analysis results are added to the recipe data to generate the final recipe.

[1959] Input: The server sends the optimal recipe to the user's device.

[1960] Specific operation: Sends an HTTP response containing recipe data to the device.

[1961] Output: The user receives the suggested recipe on their device.

[1962] Step 4: Premium Recipe Proposal

[1963] Input: Premium members can additionally input specific preferences (e.g., low salt, no oil).

[1964] Specific actions: Enter your desired criteria using the text fields or checkboxes and click the "Submit" button.

[1965] Output: The desired condition data is sent from the terminal to the server.

[1966] Input: The server generates a customized recipe taking into account your desired conditions.

[1967] What it does: Adjusts SQL queries based on desired criteria to search and extract recipe data.

[1968] Output: The customized recipe data is sent to the user's device.

[1969] Step 5: Food Photography Tips

[1970] Input: The user takes a photo of the finished dish on their device and uploads it to the server.

[1971] Specific operations: Click the "Upload a photo of the finished dish" button, select a photo, and submit it.

[1972] Output: Photo data of the finished dish is sent to the server.

[1973] Input: The server receives the uploaded photo data.

[1974] Specific operation: Decodes and saves photo data.

[1975] Output: The saved photo data is passed to the image analysis module.

[1976] Input: The server analyzes the photo data and generates recommendations on the appearance and finish of the dish.

[1977] What it does: The image analysis model evaluates the photo and generates a feedback statement.

[1978] Output: The generated advice is passed to the sentiment analysis module.

[1979] Input: The emotion engine analyzes the user's emotional state and reflects it in the advice.

[1980] Specific operation: Integrate the sentiment analysis results into the feedback sentence.

[1981] Output: A feedback statement is generated for the user to receive.

[1982] (Application example 2)

[1983] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1984] Conventional recipe recommendation systems have been unable to flexibly suggest recipes based on the user's mood or physical condition, or when the user has a variety of cooking ingredients. There is a growing demand for recipe suggestions that take into account the passenger's emotions and current health status, especially in moving environments such as autonomous vehicles. However, no systems have been available to address this demand. This creates a need for personalized recipe suggestions that efficiently use ingredients carried by the user while also responding to the passenger's emotions and health status.

[1985] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading photos of ingredients from a terminal to the server, means for the server to analyze the uploaded photos and generate an ingredient list, means for the user to select the type of dish and the number of days, means for the server to suggest a recipe based on the ingredient list and the user's selection, means for a passenger to photograph and upload ingredients for the dish using a terminal in the autonomous vehicle, means for an emotion engine to analyze the passenger's emotions and reflect them in recipe suggestions, means for the server to suggest dishes that have effects such as stress reduction based on the passenger's current emotional state, means for generating recipes that incorporate specific requests for premium members, and means for the user to additionally input desired conditions (such as carbohydrate restriction or reduced salt). This makes it possible to efficiently use the ingredients that passengers have on hand while in an autonomous vehicle and to suggest personalized recipes that are optimal for the passenger's emotions and physical condition.

[1986] "Ingredients" refer to the ingredients and raw materials used to create a dish.

[1987] "Terminal" refers to electronic devices such as smartphones, tablets, and personal computers, which are devices that are directly operated by users.

[1988] A "server" is a computer system installed in the cloud or at a remote location, and is a device that analyzes and processes data.

[1989] "Upload" means sending data from the user's terminal to the server.

[1990] "Photo" refers to image data taken by a user using the camera function of the terminal.

[1991] "Image analysis" is a method by which the server analyzes photographic data to identify materials and other elements.

[1992] The "ingredient list" is a list of cooking ingredients identified through image analysis.

[1993] A "recipe" is a detailed description of the steps and ingredients needed to prepare a particular dish.

[1994] An "emotion engine" is a software or hardware system for analyzing a user's emotional state.

[1995] "Selection information" refers to conditions such as the type of food desired by the user and the number of days.

[1996] A "Premium Member" is a registered user who has access to additional services and customization features.

[1997] "Desired conditions" are requirements that the user specifically specifies when proposing a recipe (e.g., carbohydrate restriction, reduced salt, etc.).

[1998] An "autonomous vehicle" is a vehicle that operates automatically without a human driver.

[1999] "Passenger" means a user riding in an automated driving vehicle.

[2000] "Personalization" refers to customizing something to suit the preferences and needs of an individual user.

[2001] "Feedback" refers to the reactions and opinions of users regarding the services and suggestions provided.

[2002] The system of this invention allows users to photograph and upload ingredients they have, analyze them, and suggest optimal recipes. Furthermore, by combining it with an emotion engine, it is possible to suggest personalized recipes based on the user's emotional state and physical condition. Passengers in autonomous vehicles can receive cooking suggestions and advice tailored to their individual preferences using ingredients they carry with them while traveling.

[2003] Hardware and software used

[2004] Hardware

[2005] Smartphone (Android, iOS)

[2006] Information systems in autonomous vehicles

[2007] Cloud server (AWS, GCP, etc.)

[2008] software

[2009] Image analysis algorithms (TensorFlow, OpenCV)

[2010] Emotion engine (IBM Watson, Microsoft Azure Cognitive Services)

[2011] Recipe database (MySQL, PostgreSQL)

[2012] Mobile app development frameworks (Flutter, React Native)

[2013] System Operation

[2014] The user uses a smartphone to take a photo of the ingredients they are holding, and the photo data is uploaded to a cloud server via the autonomous vehicle's in-vehicle information system. The cloud server then uses image analysis algorithms to identify the ingredients in the photo and generate an ingredients list.

[2015] The user then selects the type of dish (e.g., diet menu, hearty menu, etc.) and the number of days (e.g., 1-day supply, 3-day supply), and sends this information from the device to the server. The server combines the selection information sent by the user with the ingredient list to search for the optimal recipe, and the emotion engine analyzes the user's emotional state and reflects it in the recipe suggestions.

[2016] For premium members, users can also enter specific preferences (e.g., low sugar, low salt), and this information is also sent to the server. The server then generates and provides customized recipes to users that take these preferences into account.

[2017] Once the dish is ready, the user takes a photo of the dish with their smartphone and uploads it to the server, which analyzes the photo and generates advice about the appearance and finish.The emotion engine analyzes the user's emotional state and reflects it in the advice.

[2018] Specific examples

[2019] For example, a passenger in an autonomous vehicle can take a photo of the chicken breast, tomatoes, and broccoli they purchased that day and upload it with their smartphone. At the same time, they can select a "diet menu" and input that they are currently feeling "stressed." The server analyzes the ingredient list, and the emotion engine suggests dishes that are effective in "reducing stress" (e.g., grilled chicken breast with a balanced salad).

[2020] Prompt sentence to input to the generative AI model:

[2021] I took photos of ingredients and uploaded them. I'd like a diet menu this time. I'm also feeling stressed right now. Please suggest recipes that take stress reduction into consideration.

[2022] The system allows passengers on the move to make the most of the ingredients they have on hand and easily create healthy and delicious meals tailored to their individual desires and emotional state.

[2023] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2024] Step 1:

[2025] The user uses a smartphone to take a photo of the ingredients to be used in cooking, launches the smartphone's camera app, and presses the capture button to generate photo data.

[2026] Input: smartphone, materials

[2027] Output: Photo data

[2028] Step 2:

[2029] The device uploads the captured photo data to the cloud server, and the device's application sends the photo data via the cloud service's API.

[2030] Input: Photo data

[2031] Output: Upload photo data to cloud server

[2032] Step 3:

[2033] The server receives the uploaded photos and uses image analysis algorithms to identify materials in the photos. The server performs image analysis using TensorFlow or OpenCV.

[2034] Input: Photo data

[2035] Output: Materials list

[2036] Step 4:

[2037] The server generates a material list and saves it in the database. The analyzed material information is saved in list format.

[2038] Input: Material data

[2039] Output: Material list saved in database

[2040] Step 5:

[2041] The user selects the type of food and the number of days they want to eat it using the device application. The user taps on the selection screen to select an item, and the selection information is generated.

[2042] Input: User selection information (food type, number of days)

[2043] Output: Selection information

[2044] Step 6:

[2045] The device sends the selection information to the cloud server. The device application calls an API to send the selection information to the server.

[2046] Input: Selection information

[2047] Output: Send to server

[2048] Step 7:

[2049] The server searches the database for the best recipe based on the ingredients list and selection information. It queries the database using an SQL query.

[2050] Input: Material list, selection information

[2051] Output: Recipe candidates

[2052] Step 8:

[2053] The emotion engine analyzes the user's emotional state and reflects it in recipe suggestions. The emotion engine obtains emotional data from the user's voice and facial expressions, analyzes it, and reflects it in recipe search results.

[2054] Input: Emotion data

[2055] Output: Personalized recipe suggestions

[2056] Step 9:

[2057] The server provides personalized recipes to the user's device, where the suggested recipes are sent to an application on the device and displayed on the screen.

[2058] Input: Personalized Recipe

[2059] Output: Recipe display

[2060] Step 10:

[2061] For premium members, users can enter additional specific conditions they desire (e.g., carbohydrate restriction, reduced salt intake, etc.). Desired condition data is generated when users enter additional desired conditions into the terminal.

[2062] Input: Desired conditions

[2063] Output: Desired condition data

[2064] Step 11:

[2065] The device sends the desired conditions data to the server. The device application calls the API to send the desired conditions data to the server.

[2066] Input: Desired conditions data

[2067] Output: Send to server

[2068] Step 12:

[2069] The server generates a customized recipe based on the user's desired conditions and provides it to the user's device. The server then searches the database again, selects a recipe that meets the desired conditions, and sends it to the device.

[2070] Input: Desired conditions, material list

[2071] Output: Customized recipe

[2072] Step 13:

[2073] Once the user has completed the cooking, they take a photo of it with their smartphone and upload it to the server. They then use the smartphone's camera app to take another photo and upload it to the cloud server.

[2074] Input: Food photo

[2075] Output: Upload photos to cloud server

[2076] Step 14:

[2077] The server analyzes the uploaded photos of the finished dish, and the emotion engine analyzes the user's emotional state and reflects it in the advice. The server uses an image analysis algorithm, and the emotion engine analyzes the user's emotional data.

[2078] Input: Photos of finished dishes, user emotion data

[2079] Output: Advice

[2080] Step 15:

[2081] The server provides the generated advice to the user's terminal, where it is sent to an application on the terminal and displayed to the user.

[2082] Input: Advice

[2083] Output: Advice display

[2084] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2085] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2086] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2087] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2088] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2089] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2090] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2091] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2092] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2093] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2094] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2095] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2096] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2097] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2098] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2099] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2100] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2101] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2102] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2103] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2104] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2105] The following is further disclosed regarding the above embodiment.

[2106] (Claim 1)

[2107] A means for uploading photos of ingredients from the terminal to the server;

[2108] a means for the server to analyze the uploaded photographs and generate an ingredients list;

[2109] a means for the user to select a type of meal and number of days;

[2110] a means for the server to suggest recipes based on the ingredient list and user selections;

[2111] A system including:

[2112] (Claim 2)

[2113] A means to generate recipes incorporating specific preferences for premium members;

[2114] The system of claim 1 further comprising:

[2115] (Claim 3)

[2116] A means for analyzing uploaded photos of finished dishes and generating advice;

[2117] The system of claim 1 further comprising:

[2118] "Example 1"

[2119] (Claim 1)

[2120] means for uploading a photograph of the material from the terminal to the data processing device;

[2121] means for the data processing device to analyze the uploaded photographs and generate a materials list;

[2122] a means for a user to select a cooking type and duration;

[2123] means for the data processing device to suggest a food preparation method based on the ingredient list and the user's selection;

[2124] means for generating a customized food production recipe utilizing the generative AI model based on a user's specific preferences;

[2125] a means for analyzing the uploaded photograph of the finished food product to generate advice;

[2126] A system including:

[2127] (Claim 2)

[2128] 10. The system of claim 1, further comprising means for generating food production recipes incorporating specific preferences for premium members.

[2129] (Claim 3)

[2130] 10. The system of claim 1, further comprising means for analyzing uploaded photos of finished food products to generate recommendations regarding the finished product.

[2131] "Application Example 1"

[2132] (Claim 1)

[2133] A means for uploading photos of ingredients from the terminal to the server;

[2134] a means for the server to analyze the uploaded photographs and generate an ingredients list;

[2135] a means for the user to select a type of meal and number of days;

[2136] a means for the server to suggest recipes based on the ingredient list and user selections;

[2137] a means for listing additional ingredients for the suggested recipe;

[2138] A means of ordering missing ingredients via a food delivery service;

[2139] A system including:

[2140] (Claim 2)

[2141] A means to generate recipes incorporating specific preferences for premium members;

[2142] The system of claim 1 further comprising:

[2143] (Claim 3)

[2144] A means for analyzing uploaded photos of finished dishes and generating advice;

[2145] The system of claim 1 further comprising:

[2146] "Example 2: Combining Emotion Engines"

[2147] (Claim 1)

[2148] A means for uploading photos of ingredients from the terminal to the server;

[2149] a means for the server to analyze the uploaded photographs and generate an ingredients list;

[2150] means for the user to select a type of meal and a duration;

[2151] a means for the server to suggest recipes based on the ingredient list and user selection information;

[2152] A means for analyzing the emotional state of a user using an emotion analysis means and reflecting the result in recipe suggestions;

[2153] A system including:

[2154] (Claim 2)

[2155] A means to generate recipes incorporating specific preferences for premium members;

[2156] The system of claim 1 further comprising:

[2157] (Claim 3)

[2158] A means for analyzing uploaded photos of finished dishes and generating advice;

[2159] A means for analyzing the emotional state of the user using an emotion analysis means and reflecting the result in advice;

[2160] The system of claim 1 further comprising:

[2161] "Application example 2 when combining emotion engines"

[2162] (Claim 1)

[2163] A means for uploading photos of ingredients from the terminal to the server;

[2164] a means for the server to analyze the uploaded photographs and generate an ingredients list;

[2165] a means for the user to select a type of meal and number of days;

[2166] a means for the server to suggest recipes based on the ingredient list and user selections;

[2167] Passengers can use the self-driving vehicle's terminal to take photos and upload cooking ingredients,

[2168] The emotion engine analyzes passengers' emotions and reflects them in recipe suggestions.

[2169] A means for servers to suggest dishes that have effects such as stress reduction based on the passenger's current emotional state;

[2170] A system including:

[2171] (Claim 2)

[2172] A means to generate recipes incorporating specific preferences for premium members;

[2173] A means for users to input additional desired conditions (sugar restriction, reduced salt, etc.),

[2174] The system of claim 1 further comprising:

[2175] (Claim 3)

[2176] A means for analyzing uploaded photos of finished dishes and generating advice;

[2177] A means for the emotion engine to analyze the user's emotional state and reflect it in advice;

[2178] The system of claim 1 further comprising: [Explanation of symbols]

[2179] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for uploading photos of ingredients from the terminal to the server; a means for the server to analyze the uploaded photographs and generate an ingredients list; a means for the user to select a type of meal and number of days; a means for the server to suggest recipes based on the ingredient list and user selections; A system including:

2. A means to generate recipes incorporating specific preferences for premium members; The system of claim 1 further comprising:

3. A means for analyzing uploaded photos of finished dishes and generating advice; The system of claim 1 further comprising:

Images