System

Abstract

To provide a system that improves a user's health management.SOLUTION: A system includes means for taking a photo of a user's meal, means for analyzing ingredients contained in the dish from the taken photo, and means for generating and presenting advice, by using a prompt sentence instructing to generate the advice on the presence or absence of allergens, estimated calories, and recommended drinks or foods to be consumed together based on the analysis results, and a generative AI model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] It is difficult to judge for oneself the contents of a meal, including whether or not it contains allergens, calorie counting, and combinations with drinks and other foods. In addition, when dining with multiple people, it is difficult to suggest dishes that suit each person's preferences. [Means for solving the problem]

[0005] This system takes a photo of a meal and analyzes the ingredients contained in the dish from the photo. Based on the analysis results, it sends advice on whether or not there are any allergies, estimated calories, and recommended drinks and foods to accompany the meal. Furthermore, when dining with multiple people, it conducts a questionnaire based on the preferences of the participants and suggests dishes based on the results. These results are saved and used to suggest meals for future visits. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] When a user takes a photo of a meal, the system analyzes the ingredients contained in the dish from the photo. The analysis is performed using image recognition technology, specifically a deep learning-based neural network, to identify the type of dish and its ingredients from the photo. Based on the analysis results, the system sends the user advice on whether the dish contains any allergens, estimated calories, and recommended drinks and foods to accompany the dish.

[0029] "Example 2"

[0030] For multi-person dining, the system of the present invention conducts a survey based on the participants' preferences. The survey is conducted through an interface provided by the system, and participants input their preferences and dietary restrictions. Based on the survey results, the system recommends dishes. For example, if the survey reveals that all participants prefer fish dishes, the system will suggest a menu centered around fish dishes.

[0031] "Example 3"

[0032] The system of the present invention saves the analysis results and questionnaire results and makes meal suggestions based on those results for the next meal. The results are saved in a database within the user's profile and are referenced when making the next meal suggestion. For example, if a user ate a high-calorie meal in their last meal, the system can suggest a low-calorie meal for the next meal, taking into account the user's dietary balance.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The user takes a photo of their meal.

[0036] Step 2: The system analyzes the ingredients in the dish from the photo using a deep learning-based neural network.

[0037] Step 3: Based on the analysis results, the system will send the user advice on whether they have any allergies, estimated calories, and recommended drinks and foods to ingest together.

[0038] "Example 2"

[0039] Step 1: For multi-person dining, the system surveys participants based on their preferences.

[0040] Step 2: Participants complete a questionnaire, including their preferences and dietary restrictions.

[0041] Step 3: Based on the survey results, the system will suggest recommended dishes.

[0042] "Example 3"

[0043] Step 1: The system saves the analysis results and survey results.

[0044] Step 2: The saved results will be referenced when suggesting meals from the next time onwards.

[0045] Step 3: The next time a meal recommendation is made, the system will use the saved results to make the recommendation.

[0046] Example 1

[0047] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0048] Conventional meal management systems require users to manually enter meal details, which is time-consuming and often lacks accuracy. They also struggle to provide comprehensive meal management, such as checking for allergens, calculating calories, and providing advice on accompanying foods and drinks. Furthermore, when dining with multiple people, they are unable to provide suggestions that take into account the preferences and allergy information of each participant. To address these issues, there is a need for a system that allows users to easily manage meal details and provides accurate allergy information, calorie calculations, and appropriate advice.

[0049] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0050] In this invention, the server includes a means for acquiring an image of a meal, a means for analyzing the ingredients contained in the dish from the acquired image, a means for determining the presence or absence of allergens based on the analysis results, a means for calculating estimated calories based on the analysis results, and a means for providing advice on recommended foods and beverages to ingest based on the analysis results. This allows users to easily manage their meal contents and receive accurate allergy information, calorie calculations, and appropriate advice simply by taking a photo of their meal. Furthermore, in the case of a meal with multiple people, the server further includes a means for conducting a survey based on the preferences of the participants and a means for suggesting recommended dishes based on the survey results, thereby enabling suggestions that take into account the preferences and allergy information of all participants. Furthermore, the server includes a means for saving the analysis results and survey results and making meal suggestions based on those results for future meals, facilitating ongoing dietary management.

[0051] "Means for acquiring images of meals" refers to devices or software that allow a user to take a photo of a meal and send the image data to the system.

[0052] "Means for analyzing the ingredients contained in a dish from an acquired image" refers to devices or software that use image recognition technology to identify the type of dish and its ingredients from a photograph of a meal.

[0053] "Means for determining whether or not an allergen is present" refers to devices or software that compares the analyzed material information with the allergy information registered by the user to confirm whether or not an allergen is present.

[0054] "Means for calculating estimated calories" refers to devices or software for calculating the total calories of a dish based on analyzed ingredient information.

[0055] "Means for providing advice on foods and beverages to be consumed together" refers to devices and software that suggest foods and beverages that go well with a dish based on the analysis results.

[0056] "Means for taking surveys" refers to devices or software used to collect information about preferences and allergies of participants at group meals.

[0057] "Means for suggesting recommended dishes" refers to devices or software that make recommendations based on the survey results, taking into account the preferences and allergy information of all participants.

[0058] "Means for saving analysis results and survey results" refers to devices and software for saving analyzed material information and survey results in a database or the like.

[0059] "Means for making meal suggestions for the next time and onwards" refers to devices or software for making meal suggestions for the next time and onwards based on the saved analysis results and survey results.

[0060] This invention is a system that allows a user to take a photo of a meal and analyzes the ingredients contained in the dish from the photo. The system includes means for acquiring an image of the meal, means for analyzing the ingredients contained in the dish from the acquired image, means for determining the presence or absence of allergens based on the analysis results, means for calculating estimated calories based on the analysis results, and means for providing advice on recommended foods and drinks to ingest along with the meal based on the analysis results.

[0061] Hardware and software used

[0062] Users take photos of their meals using a smartphone or camera-equipped device. The device then uploads the photos to the server. The server receives the uploaded photos and uses deep learning libraries such as TENSORFLOW (registered trademark) and PyTorch to perform image analysis. The analysis results are stored in a database and notified to the user.

[0063] Specific examples

[0064] A user orders "curry rice" at a restaurant and takes a photo of it with their smartphone. The device uploads the photo to the server. The server receives the photo and performs image analysis using TensorFlow. As a result of the analysis, ingredients such as "rice, curry sauce, meat, and vegetables" are identified. The server compares this with the allergy information registered by the user to determine whether or not any allergens are present. For example, if the user has a "nut allergy," the server checks whether the ingredients contain nuts. Next, the server calculates the estimated calorie content of the dish based on the ingredients and displays "700 kcal." The server then generates advice such as "Lassi goes well with curry rice" and notifies the user.

[0065] Prompt Sentence Examples

[0066] "Analyze a user-taken photo of spaghetti bolognese and identify the ingredients. If the user has a dairy allergy, display a warning and provide food and drink recommendations along with estimated calories."

[0067] In this way, users can easily manage their meal contents, receive accurate allergy information, calorie calculations, and appropriate advice simply by taking photos of their meals. Furthermore, for meals with multiple people, including a means for conducting a survey based on the participants' preferences and a means for proposing recommended dishes based on the survey results makes it possible to make suggestions that take into account the preferences and allergy information of all participants. Furthermore, including a means for saving the analysis results and survey results and making future meal suggestions based on those results makes ongoing dietary management easier.

[0068] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0069] Step 1:

[0070] The user takes a photo of the meal.

[0071] A user takes a photo of a meal using a smartphone or other device with a camera. For example, a user orders "curry rice" at a restaurant and takes a photo of it. The input is the photo of the meal, and the output is the captured image data.

[0072] Step 2:

[0073] The device uploads the photo to the server.

[0074] The device sends the captured photos to a server via an Internet connection. The input is the captured image data, and the output is the image data uploaded to the server.

[0075] Step 3:

[0076] The server receives the photo and performs image analysis.

[0077] The server receives the photos sent from the device. It performs image analysis on the received photos using deep learning libraries such as TensorFlow and PyTorch. The input is the uploaded image data, and the output is the analyzed type of food and ingredient information. Specifically, the neural network model analyzes the photo and identifies "curry rice" and its ingredients.

[0078] Step 4:

[0079] The server identifies the material based on the analysis results.

[0080] The server identifies the type of food and its ingredients in the photo based on the results of image analysis. The input is the analyzed information on the type of food, and the output is the identified ingredient information. For example, if "curry rice" is identified, ingredients such as "rice, curry sauce, meat, and vegetables" will be listed.

[0081] Step 5:

[0082] The server checks for the presence of allergens.

[0083] The server compares the identified ingredients with the allergy information registered by the user. The input is the identified ingredient information and the user's allergy information, and the output is the determination result of whether or not the allergen is present. For example, if the user has a "nut allergy," it checks whether the ingredients contain nuts.

[0084] Step 6:

[0085] The server calculates the estimated calories.

[0086] The server calculates the estimated calories of a dish based on the identified ingredients. The input is the identified ingredient information, and the output is the calculated estimated calories. For example, in the case of "curry rice," the calories of each ingredient are added together to calculate "700 kcal."

[0087] Step 7:

[0088] The server then generates advice on recommended drinks and foods to consume.

[0089] Based on the analysis results, the server generates advice on recommended drinks and foods to ingest. The input is the analysis results and the identified ingredient information, and the output is the generated advice. For example, it generates advice such as "Lassi goes well with curry rice."

[0090] Step 8:

[0091] The server sends the analysis results and advice to the user.

[0092] The server sends the analysis results and advice to the user. The input is the analysis results and generated advice, and the output is a notification sent to the user. The user can check the results on the screen of their smartphone or device.

[0093] (Application example 1)

[0094] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] In today's modern diet, it is important for users to accurately understand what they are eating and manage their health. However, when eating out or using delivery services, it is difficult to understand the ingredients, allergens, and calorie information contained in the food. There is also a lack of systems that suggest appropriate drinks or additional dishes based on the user's preferences and health status. This makes it difficult for users to manage their health and increases the risk of allergic reactions and overdose.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0097] In this invention, the server includes means for taking a photo of a meal, means for analyzing the ingredients contained in the dish from the taken photo, means for sending information on the presence or absence of allergens, estimated calories, and advice on recommended drinks and foods to accompany the dish based on the analysis results, means for taking a photo of a dish ordered by a user, analyzing the ingredients contained in the dish from the photo, and providing information on the presence or absence of allergens and calories, and means for suggesting recommended drinks and additional dishes to accompany the dish based on the user's preferences and health condition. This allows the user to accurately understand the contents of their meal and manage their health.

[0098] "Means for taking photos of meals" refers to devices or functions that allow users to take photos of meals.

[0099] "Means for analyzing ingredients contained in a dish from a photograph" refers to a device or function for identifying ingredients contained in a dish based on a photograph.

[0100] "Means for sending advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to ingest based on the analysis results" refers to devices or functions that provide information on the presence or absence of allergens and calories based on the analysis results, and recommend appropriate drinks and foods.

[0101] "Means for taking a photo of the dish ordered by the user, analyzing the ingredients contained in the dish from the photo, and providing calorie information and the presence or absence of allergens" refers to a device or function that takes a photo of the dish ordered by the user, analyzes the ingredients based on the photo, and provides calorie information and the presence or absence of allergens.

[0102] "Means for suggesting drinks and additional dishes to be consumed together based on the user's preferences and health condition" refers to a device or function for suggesting appropriate drinks and additional dishes taking into consideration the user's preferences and health condition.

[0103] The system for implementing this invention allows users to take a photo of a meal and analyze the ingredients contained in the dish from the photo. The system is realized as an application installed on a mobile device such as a smartphone or tablet.

[0104] The server uses TensorFlow and the ResNet50 model for image recognition. When a user takes a photo of a meal, the photo is sent to the server. The server preprocesses the photo and inputs it into the ResNet50 model to identify the ingredients contained in the dish. Based on the identified ingredients, the server provides information on the presence or absence of allergens and calorie content. It also suggests recommended drinks and additional dishes to accompany the meal based on the user's preferences and health status.

[0105] Specifically, the server performs the following steps: First, it receives a photo of the meal taken by the user and preprocesses the image. Next, it inputs the preprocessed image into a ResNet50 model to identify the ingredients of the dish. Based on the identified ingredients, it checks for the presence of allergens and calculates calorie information. Furthermore, it takes into account the user's preferences and health condition to suggest appropriate drinks and additional dishes.

[0106] The hardware used is a mobile device such as a smartphone or tablet, and the software uses the TensorFlow library and ResNet50 model, allowing users to accurately track their diet and manage their health.

[0107] For example, if a user takes a photo of a pizza, the server can identify the ingredients in the pizza, check for allergens (e.g., gluten), provide calorie information for the pizza, and suggest a drink (e.g., cola) to accompany it.

[0108] Example prompts to input to a generative AI model:

[0109] Analyze photos of meals taken by users to identify ingredients, identify allergens, provide calorie information, and suggest companion drinks and additional dishes.

[0110] In this way, a food delivery service application can be realized that provides information that takes into consideration the user's health and preferences.

[0111] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0112] Step 1:

[0113] The user takes a photo of the meal.

[0114] Input: A photo of a meal taken by the user on their smartphone or tablet.

[0115] Output: Photo data of the food taken.

[0116] How it works: The user takes a photo of their meal using the camera on their smartphone or tablet, which is then saved to their device.

[0117] Step 2:

[0118] The device sends the photograph to the server.

[0119] Input: Photo data of the food taken.

[0120] Output: The photo data sent to the server.

[0121] Specific operation: The device sends the captured photo data to the server via the Internet using a communication protocol such as an HTTP request.

[0122] Step 3:

[0123] The server preprocesses the received photos.

[0124] Input: Photo data sent to the server.

[0125] Output: Preprocessed image data.

[0126] Specific operation: The server performs preprocessing such as resizing and normalizing the received photo data, and converts it into a format suitable for the image recognition model.

[0127] Step 4:

[0128] The server inputs the preprocessed images into a ResNet50 model to identify the ingredients of the dish.

[0129] Input: Preprocessed image data.

[0130] Output: A list of ingredients for the identified dish.

[0131] How it works: The server inputs the preprocessed image data into the ResNet50 model, which analyzes the image and identifies the ingredients in the dish. The identified ingredients are then output in a list format.

[0132] Step 5:

[0133] The server checks for the presence of allergens based on the identified ingredients and calculates calorie information.

[0134] Input: A list of ingredients for the identified dish.

[0135] Output: Allergen information and calorie information.

[0136] Specific operation: The server refers to the specified ingredient list, checks for allergens, and calculates the calorie count of the entire dish based on the calorie information of each ingredient.

[0137] Step 6:

[0138] The server suggests recommended drinks and additional dishes to accompany the meal based on the user's preferences and health status.

[0139] Input: Ingredient list of identified dishes, whether or not they contain allergens, calorie information, and user preferences and health data.

[0140] Output: A list of recommended drinks and additional dishes.

[0141] Specific operation: The server refers to the user's preferences and health data, and suggests suitable drinks and additional dishes based on the identified ingredients and calorie information. The suggestions are output in list format.

[0142] Step 7:

[0143] The server sends the analysis results and suggestions to the user.

[0144] Input: allergen availability, calorie information, and a list of recommended drinks and extra dishes.

[0145] Output: Analysis results and suggestions sent to the user.

[0146] How it works: The server compiles the analysis results and recommendations and sends them to the user's device, where they can view the results on their smartphone or tablet.

[0147] Example 2

[0148] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0149] Conventional meal recommendation systems have difficulty in making recommendations that fully take into account participants' preferences and dietary restrictions, making it difficult to choose a menu that satisfies everyone, especially when dining with multiple people. Furthermore, analysis results and survey results could not be used to make recommendations in the future, which meant that users had to enter the same information each time.

[0150] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0151] In this invention, the server includes means for taking photos of meals, means for analyzing the ingredients contained in the dishes from the photos, means for sending information on the presence or absence of allergies, estimated calories, and advice on recommended drinks and foods to ingest based on the analysis results, means for taking a survey based on participants' preferences, means for saving the survey results, means for analyzing the saved survey results, means for suggesting recommended dishes based on prompts using a generative AI model, and means for displaying the recommendation results. This makes it possible to suggest optimal dishes taking into account the preferences and dietary restrictions of all participants, and past data can be used for future meal suggestions.

[0152] "Means for taking photos of meals" refers to devices or software that allow a user to take photos of meals and send the image data to the system.

[0153] "Means for analyzing ingredients contained in a dish from a photograph" refers to devices or software that use image analysis technology to identify ingredients contained in a dish from a photograph of the food.

[0154] "Means of sending advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to ingest based on the analysis results" refers to devices and software that calculate the presence or absence of allergens and calories based on analyzed ingredient information, and provide users with advice on appropriate drinks and foods.

[0155] "Means for surveying participants based on their preferences" refers to an interface or software used to collect dietary preferences and restrictions from multiple participants.

[0156] "Means for storing survey results" refers to devices or software for storing collected survey data in a storage device such as a database.

[0157] "Means for analyzing stored survey results" refers to any device or software used to analyze stored survey data and understand participants' preferences and limitations.

[0158] "Means for using a generative AI model to suggest recommended dishes based on a prompt sentence" refers to a device or software that inputs a prompt sentence into a generative AI model and generates optimal dish suggestions as a result.

[0159] "Means for displaying the suggestion results" refers to devices or software for visually displaying the dish suggestion results obtained from the generative AI model to the user.

[0160] The present invention is a system for proposing optimal dishes for a group meal, taking into consideration the preferences and dietary restrictions of the participants. A specific embodiment of this system will be described below.

[0161] The server first provides a questionnaire interface that users can access. This interface is built using HTML, CSS, and JavaScript (registered trademark) and runs on a web browser. Users access this interface using their devices (PCs or smartphones) and enter their preferences and dietary restrictions.

[0162] When the user has completed filling out the questionnaire form, the device sends the data to the server, which processes the received data using a Python script and stores it in a database (e.g., MySQL (registered trademark) or PostgreSQL).

[0163] The server then analyzes the stored survey results using a generative AI model (e.g., OpenAI's GPT-4®). The server inputs the survey results as prompts into the generative AI model to generate optimal dish suggestions.

[0164] Specifically, the following prompt sentences are input to the generative AI model:

[0165] "Based on the results of the following survey, please suggest a dish that will satisfy all participants. Survey results: Participant A likes fish dishes, Participant B is a vegetarian, and Participant C does not like spicy food."

[0166] Based on this prompt, the generative AI model will return a suggestion, such as "grilled fish and vegetable salad with a mild dressing." The server will then display this suggestion to the user, who can then view it through their browser and choose a dish based on it.

[0167] Furthermore, the server stores the analysis results and survey results and can use them to make meal suggestions for future visits, eliminating the need for users to enter the same information each time and enabling more efficient suggestions of the most suitable dishes.

[0168] For example, suppose participants A, B, and C at a dinner party have the following preferences and restrictions:

[0169] Participant A: I like fish dishes.

[0170] Participant B: Vegetarian

[0171] Participant C: I don't like spicy food.

[0172] In this case, the user fills in a questionnaire form with their preferences and restrictions. The device sends the data to a server, which stores and analyzes it. The generative AI model is then given the following prompt:

[0173] "Based on the results of the following survey, please suggest a dish that will satisfy all participants. Survey results: Participant A likes fish dishes, Participant B is a vegetarian, and Participant C does not like spicy food."

[0174] The generative AI model returns suggestions such as "grilled fish with vegetable salad, served with a mild dressing," which the server displays to the user, who can then choose a dish based on the suggestions.

[0175] In this way, the invention allows for optimal dish suggestions that take into account the preferences and dietary restrictions of all participants.

[0176] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0177] Step 1: Providing a survey interface

[0178] The server provides a survey interface that users can access. This interface is built using HTML, CSS, and JavaScript and runs on a web browser. Users access this interface using their devices (PCs or smartphones).

[0179] Input: User's access request

[0180] Output: Display the survey form

[0181] Specific operation: The server provides the HTML file of the questionnaire form through the web server and displays it in the user's browser.

[0182] Step 2: Enter and submit survey data

[0183] Users fill out a questionnaire about their preferences and dietary restrictions, then click the submit button.

[0184] Input: User preferences and dietary restrictions

[0185] Output: Send survey data

[0186] Specific operation: The user enters text or options into form fields and clicks the submit button. The device uses JavaScript to collect the form data and sends it to the server as an HTTP POST request.

[0187] Step 3: Save your survey data

[0188] The server stores the received survey data in a database using Python scripts and a database management system.

[0189] Input: Survey data

[0190] Output: Save to database

[0191] What it does: The server processes the received data using a Python framework, establishes a database connection, executes SQL queries, and saves the data.

[0192] Step 4: Analyze the survey data

[0193] The server analyzes the stored survey data using a generative AI model.

[0194] Input: Saved survey data

[0195] Output: prompt statement

[0196] Specific operation: The server retrieves the survey data from the database using a Python script and generates a prompt sentence.

[0197] Step 5: Food recommendations

[0198] The generative AI model suggests dishes based on the prompt, and the server receives the suggestions.

[0199] Input: prompt statement

[0200] Output: Food recommendations

[0201] Specific operation: The server sends a prompt sentence to the generative AI model, and the generative AI model generates a suggestion result and returns it to the server.

[0202] Step 6: Viewing the Suggestion Results

[0203] The server displays the recommendations received from the generative AI model to the user, who can then choose a dish based on the recommendations.

[0204] Input: Food recommendations

[0205] Output: Display of proposal results

[0206] Specific operation: The server formats the proposal results in HTML format and provides them to the user as a web page. The user can then view the proposal results through their browser.

[0207] (Application example 2)

[0208] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0209] Conventional meal recommendation systems have difficulty considering individual user preferences and dietary restrictions, making it difficult to recommend dishes that will satisfy everyone, especially when dining with multiple people. They also lack a mechanism for effectively utilizing user preference data and reflecting it in future meal recommendations. Furthermore, there was a need for a simple interface that can be used with smartphones.

[0210] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0211] In this invention, the server includes means for acquiring images of food, means for analyzing the ingredients contained in the food from the acquired images, means for providing, based on the analysis results, the presence or absence of allergens, estimated calories, and advice on recommended beverages and foods to ingest together, means for conducting a survey based on the preferences of participants in the case of a meal with multiple people, means for suggesting recommended dishes based on the survey results, means for saving the survey results and using those results to suggest meals for the next time and thereafter, means for collecting user preferences and dietary restrictions through an application installed on a smartphone and suggesting optimal dishes based on shared preferences, and means for analyzing user preference data using a generative AI model and generating prompt sentences, thereby making it possible to suggest dishes that will satisfy everyone in a meal with multiple people.

[0212] The "means for acquiring an image of a meal" is a function that allows a user to take a photo of a meal and send the image data to the system.

[0213] The "means for analyzing the ingredients contained in a dish from an acquired image" is a function that uses image analysis technology to identify the ingredients contained in a dish from a photograph of the food.

[0214] "Means for providing advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to ingest together" is a function that calculates the presence or absence of allergens and estimated calories based on analyzed ingredient information, and also provides users with advice on appropriate drinks and foods.

[0215] "In the case of a meal with multiple people, a means of conducting a survey based on the preferences of the participants" is a function that collects information about the preferences and dietary restrictions of multiple users in the form of a survey.

[0216] "Means for suggesting recommended dishes based on survey results" is a function that analyzes collected survey results and suggests dishes that will satisfy everyone.

[0217] "Means for saving survey results and making meal suggestions based on those results" is a function that saves the collected survey results in a database and uses them to make meal suggestions for the next time and onwards.

[0218] "A means of collecting user preferences and dietary restrictions through an application installed on a smartphone and suggesting optimal dishes based on shared preferences" refers to a function that uses a smartphone application to allow users to input their preferences and dietary restrictions, finds shared preferences based on that data, and suggests optimal dishes.

[0219] "Means for analyzing user preference data using a generative AI model and generating prompt sentences" refers to a function that uses a generative AI model to analyze user preference data and generate appropriate prompt sentences based on the results.

[0220] As an embodiment of the present invention, the following system is constructed.

[0221] System Configuration

[0222] Hardware:

[0223] Smartphone

[0224] server

[0225] Database (SQLite)

[0226] software:

[0227] Smartphone application (iOS / ANDROID (registered trademark))

[0228] Server-side programs (Python, Flask)

[0229] Image analysis software (OpenCV)

[0230] Generative AI models (such as GPT-3 (registered trademark))

[0231] Processing flow

[0232] 1. How to get food images:

[0233] Users take photos of their meals using a smartphone app, and the images are sent to a server via the app.

[0234] 2. A method for analyzing the ingredients contained in a dish from the captured image:

[0235] The server analyzes the received images using image analysis software (OpenCV) and identifies the ingredients contained in the dish.

[0236] 3. Means of providing advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to ingest together:

[0237] Based on the analysis results, the system calculates the presence or absence of allergens and estimated calories, and provides users with advice on appropriate beverages and foods.

[0238] 4. For multi-person dining, survey participants based on their preferences:

[0239] For meals with multiple people, a survey about each participant's preferences and dietary restrictions is conducted via a smartphone app.

[0240] 5. How to suggest dishes based on survey results:

[0241] The server analyzes the collected survey results and suggests dishes that will satisfy everyone.

[0242] 6. A means to store survey results and use them to make future meal recommendations:

[0243] The survey results are stored in a database (SQLite) and will be used to make meal suggestions from next time onwards.

[0244] 7. A means of collecting user preferences and dietary restrictions through an application installed on a smartphone and suggesting the best dishes based on shared preferences:

[0245] The smartphone app collects users' preferences and dietary restrictions and suggests the best dishes based on shared tastes.

[0246] 8. Using a generative AI model to analyze user preference data and generate prompts:

[0247] The server uses a generative AI model (such as GPT-3) to analyze user preference data and generate appropriate prompts based on the results.

[0248] Specific examples

[0249] For example, if participant A likes "sushi," participant B likes "sushi," and participant C likes "sushi," the system will suggest "sushi." Allergen and calorie information will also be provided.

[0250] Example prompt sentence:

[0251] "You can enter your preferences and dietary restrictions. For example, your favorite food is 'sushi,' your least favorite food is 'green peppers,' and you have no dietary restrictions. The system will then use this information to suggest dishes that will please everyone."

[0252] In this way, users can easily have meals that suit their preferences suggested.

[0253] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0254] Step 1:

[0255] Users take photos of their meals using a smartphone app.

[0256] Input: Food photo

[0257] Output: Captured image data

[0258] Specific operation: The user takes a photo of a meal using the camera function of their smartphone and saves the image data in the application.

[0259] Step 2:

[0260] The terminal transmits the captured image data to the server.

[0261] Input: Captured image data

[0262] Output: Image data sent to the server

[0263] Specific operation: The smartphone app calls an API to upload image data to the server.

[0264] Step 3:

[0265] The server analyzes the received images using image analysis software (OpenCV) and identifies the ingredients contained in the dish.

[0266] Input: Image data sent to the server

[0267] Output: Parsed ingredient information

[0268] Specific operation: The server uses OpenCV to analyze the image and executes an algorithm to identify the ingredients contained in the dish.

[0269] Step 4:

[0270] Based on the analysis results, the server calculates the presence or absence of allergens and estimated calories, and provides the user with advice on appropriate drinks and foods.

[0271] Input: Parsed ingredient information

[0272] Output: Allergens, estimated calories, drink and food recommendations

[0273] Specific operation: The server checks for the presence of allergens based on the ingredient information, calculates calories, and generates advice on appropriate drinks and foods.

[0274] Step 5:

[0275] If dining with multiple people, the device will conduct a survey about each participant's preferences and dietary restrictions.

[0276] Input: Information about each participant's preferences and dietary restrictions

[0277] Output: Survey results

[0278] How it works: The smartphone app displays a survey form where each participant enters their preferences and dietary restrictions.

[0279] Step 6:

[0280] The server analyzes the collected survey results and suggests dishes that will satisfy everyone.

[0281] Input: Survey results

[0282] Output: Recommended dishes

[0283] What it does: The server analyzes the survey results, identifies common preferences, and runs an algorithm to suggest the best dishes based on those preferences.

[0284] Step 7:

[0285] The server stores the survey results in a database (SQLite) and uses them to make meal suggestions for future visits.

[0286] Input: Survey results

[0287] Output: Survey results stored in a database

[0288] Specific operation: The server saves the survey results in an SQLite database and performs database operations to use them for future proposals.

[0289] Step 8:

[0290] The server uses a generative AI model (such as GPT-3) to analyze user preference data and generate appropriate prompts based on the results.

[0291] Input: User preference data

[0292] Output: prompt statement

[0293] Specific operation: The server calls the generative AI model, analyzes the user's preference data as input, and generates an appropriate prompt sentence.

[0294] Example prompt sentence:

[0295] "You can enter your preferences and dietary restrictions. For example, your favorite food is 'sushi,' your least favorite food is 'green peppers,' and you have no dietary restrictions. The system will then use this information to suggest dishes that will please everyone."

[0296] Example 3

[0297] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0298] Conventional meal recommendation systems have the problem that it is difficult to make optimal meal recommendations for each individual user because they do not fully utilize the user's past eating history or questionnaire results. In addition, they do not take into consideration the balance of meals, and therefore cannot contribute to the user's health management. Furthermore, when dining with multiple people, there is the problem that they cannot make recommendations based on the preferences of all participants.

[0299] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0300] In this invention, the server includes a means for inputting the results of a questionnaire about the user's meals, a means for transmitting the inputted results of the questionnaire to the server, a means for storing the received results of the questionnaire in a database, a means for generating meal suggestions for the next meal or later based on the stored data, and a means for displaying the generated meal suggestions to the user. This enables personalized meal suggestions based on the user's past meal history and the questionnaire results, contributing to the user's health management. Furthermore, even in a multi-person meal, suggestions based on the preferences of all participants can be made.

[0301] "Means for taking photos of meals" refers to devices or functions that allow users to take photos of meals.

[0302] "Means for analyzing ingredients contained in a dish from a photograph" refers to a device or function for analyzing a photograph and identifying the ingredients contained in a dish.

[0303] "Means of sending advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to ingest based on the analysis results" refers to devices and functions that determine the presence or absence of allergens and estimated calories based on the analysis results, and recommend appropriate drinks and foods.

[0304] The term "means for inputting the results of a questionnaire about the user's diet" refers to a device or function that allows the user to answer a questionnaire about diet and input the results.

[0305] "Means for transmitting the entered survey results to the server" refers to a device or function for transmitting the survey results entered by the user to the server.

[0306] "Means for storing the survey results received by the server in a database" refers to a device or function for storing the survey results received by the server in a database.

[0307] "Means for generating meal suggestions for the next time and thereafter based on the stored data" refers to a device or function for generating meal suggestions for the next time and thereafter based on the data stored in the database.

[0308] "Means for displaying the generated meal suggestions to the user" refers to a device or function for displaying the generated meal suggestions to the user.

[0309] "Means for conducting a survey based on the preferences of participants when dining with multiple people" refers to a device or function for conducting a survey based on the preferences of participants when dining with multiple people.

[0310] "Means for suggesting recommended dishes based on survey results" refers to devices or functions for suggesting recommended dishes based on survey results.

[0311] The present invention is a system for suggesting meals for the next time and thereafter based on the results of a questionnaire and analysis of a user's meals. A specific embodiment of this system will be described below.

[0312] Hardware and software used

[0313] Hardware

[0314] Device: A device used by users to input meal information, such as a smartphone, tablet, or computer.

[0315] Server: A computer system that receives, stores, and analyzes data sent by users.

[0316] Database: A storage system connected to the server for storing users' dietary information and survey results.

[0317] software

[0318] Survey entry application: An application that allows users to fill out surveys about their diet.

[0319] Data transmission module: A software module for transmitting data from a terminal to a server.

[0320] Database Management System: Software that allows a server to store and manage data.

[0321] Generative AI model: An artificial intelligence model that generates next meal suggestions based on the user's past eating history and survey results.

[0322] System Operation

[0323] User Input

[0324] The user answers a questionnaire about their diet using a questionnaire entry application on the terminal. For example, the user enters, "I had a hamburger and fries for lunch today." This information is recorded as the user's diet history.

[0325] Sending data

[0326] The terminal transmits the meal information entered by the user to the server via the data transmission module. At this time, the terminal converts the data into an appropriate format and transmits it to the server. For example, data such as "Lunch: Hamburger and fries" is transmitted to the server.

[0327] Data storage

[0328] The server uses a database management system to store the received meal information in a database. When storing the information, metadata such as the user ID and date and time are also stored. For example, the information is stored in the database in the format "User ID: 12345, Lunch: Hamburger and fries, Date and time: 2023-10-01."

[0329] Next meal suggestion

[0330] The next time the server makes a meal suggestion, it references the database and generates a new meal suggestion based on the user's past eating history. For example, if the user previously ate a high-calorie meal, the server uses the generative AI model to suggest a low-calorie meal.

[0331] Submit a proposal

[0332] The server sends the generated meal suggestions to the device, for example, a message saying "I recommend salad and soup for your next meal."

[0333] View Suggestions

[0334] The device displays the meal suggestions received from the server to the user. For example, a message such as "We recommend salad and soup for your next meal" may be displayed on the device screen.

[0335] Examples and prompts

[0336] Specific examples

[0337] 1. The user types into the terminal, "I had a hamburger and fries for lunch today."

[0338] 2. The device sends this information to the server.

[0339] 3. The server stores this information in a database.

[0340] 4. When suggesting your next meal, the server will refer to the database and suggest, "We recommend salad and soup for your next meal."

[0341] 5. The device displays this suggestion to the user.

[0342] Prompt Sentence Examples

[0343] "Please explain the process of the system that suggests a low-calorie meal for the next meal if the user had a high-calorie meal in the previous meal." The flow of the specific process in Example 3 will be explained using Figure 15.

[0344] Step 1:

[0345] The user answers a questionnaire about meals using a questionnaire input application on the terminal.

[0346] Input: The user types, "I had a hamburger and fries for lunch today."

[0347] Output: The survey results are saved to your device.

[0348] Specific actions: The user opens the application, enters meal details into the text box, and presses the submit button.

[0349] Step 2:

[0350] The terminal transmits the meal information input by the user to the server via the data transmission module.

[0351] Input: Survey results (e.g., "Lunch: Burger and fries").

[0352] Output: Meal information is sent to the server.

[0353] Specific operation: The terminal converts the survey results into JSON format and sends them to the server using an HTTP request.

[0354] Step 3:

[0355] The server stores the received meal information in a database using a database management system.

[0356] Input: Meal information received by the server (e.g., "Lunch: hamburger and fries").

[0357] Output: Meal information stored in a database.

[0358] Specific operation: The server analyzes the received data, adds metadata such as the user ID and date and time, and saves it in the database.

[0359] Step 4:

[0360] The next time the server makes a meal suggestion, it references the database and generates a new meal suggestion based on the user's past meal history.

[0361] Input: Past dietary history stored in a database.

[0362] Output: Generated meal suggestions (e.g., "I recommend salad and soup for your next meal").

[0363] Specific operation: The server uses the generative AI model to analyze past dietary data and generate suggestions that take into account the user's health condition and dietary balance.

[0364] Step 5:

[0365] The server transmits the generated meal suggestions to the terminal.

[0366] Input: Generated meal suggestions (e.g., "I recommend salad and soup for your next meal").

[0367] Output: Meal suggestions are sent to the device.

[0368] Specific operation: The server converts the generated proposal into JSON format and sends it to the device using an HTTP response.

[0369] Step 6:

[0370] The terminal displays the meal suggestions received from the server to the user.

[0371] Input: Meal suggestions received from the server (e.g., "I recommend salad and soup for your next meal").

[0372] Output: Meal suggestions displayed to the user.

[0373] Specific operation: The device analyzes the received data and displays it on the user interface, allowing the user to check the suggestions on the screen.

[0374] (Application example 3)

[0375] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0376] Conventional meal recommendation systems have the problem that they are unable to fully utilize the user's dietary history and survey results, making it difficult to make individually optimized meal recommendations. Furthermore, since the system does not take into account the user's dietary balance, there is also the problem of inadequate health management. Furthermore, there are cases where the suggested meal menu is not displayed properly to the user, resulting in a poor user experience.

[0377] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0378] In this invention, the server includes: a means for taking a photo of a user's meal; a means for analyzing the ingredients contained in the dish from the photograph; a prompt for generating advice regarding the presence or absence of allergens, estimated calories, and recommended accompanying drinks or foods based on the analysis results; and a means for generating and presenting the advice using a generative AI model. The server also includes a means for storing the user's diet history and questionnaire results in a database; a prompt for analyzing the stored diet history or questionnaire results, evaluating the user's dietary balance, and suggesting a next meal; and a means for presenting the next meal suggestion using the generative AI model. This enables individually optimized meal suggestions based on the user's diet history and questionnaire results, improving the user's health management and improving the user experience by displaying appropriate meal menus.

[0379] "Means for taking photos of meals" refers to devices or functions that allow users to take photos of meals.

[0380] "Means for analyzing ingredients contained in a dish from a photograph" refers to a device or function for analyzing a photograph of a meal and identifying the ingredients and components contained therein.

[0381] "Means for sending advice on the presence or absence of allergens, estimated calories, and recommended foods and drinks to ingest based on the analysis results" refers to devices and functions that estimate the presence or absence of allergens and calories based on analyzed data, and provide advice on appropriate foods and drinks.

[0382] "Means for saving a user's dietary history and survey results in a database" refers to a device or function for saving information about the meals a user has eaten in the past and survey results in a database.

[0383] The "means for analyzing stored data and evaluating the user's dietary balance" refers to a device or function for analyzing stored dietary history and questionnaire results and evaluating the user's dietary balance.

[0384] The "means for suggesting the next meal based on the analysis results" refers to a device or function for suggesting the next meal based on the analyzed data.

[0385] The "means for displaying a suggested meal menu to a user" refers to a device or function for visually displaying a suggested meal menu to a user.

[0386] A system for implementing the present invention has the following configuration.

[0387] System Configuration

[0388] 1. A way to take photos of your food

[0389] Users take photos of their meals using their smartphones or smart glasses, which allows the contents of the meal to be captured as digital data.

[0390] 2. A method for analyzing ingredients contained in a dish from a photograph

[0391] The server receives the photographs and uses image analysis software (e.g., OpenCV) to identify the ingredients contained in the dish, allowing for the extraction of meal composition information.

[0392] 3. A method to send advice on whether or not you have any allergies, estimated calories, and recommended drinks and foods to ingest based on the analysis results

[0393] The server uses the analysis results to estimate the presence or absence of allergens and calories, and generates appropriate food and drink recommendations, referencing nutritional information stored in a database.

[0394] 4. A means to store user's meal history and survey results in a database

[0395] The server stores the user's meal history and survey results in a database (e.g., SQLite), allowing past data to be accumulated and used for future recommendations.

[0396] 5. A means of analyzing stored data and assessing the user's dietary balance

[0397] The server analyzes the stored data and evaluates the user's dietary balance using a data analysis library (e.g., Pandas).

[0398] 6. A method to suggest your next meal based on the analysis results

[0399] The server then uses the analysis results to suggest the next meal, and uses a generative AI model (e.g., TensorFlow) to generate the optimal meal menu for the user.

[0400] 7. How to display meal suggestions to users

[0401] The proposed meal menu is displayed on the user's device (smartphone or smart glasses) using a mobile app development framework (e.g., Flutter (registered trademark)).

[0402] Specific examples

[0403] 1. Data Collection

[0404] Users take a photo of their meal with their smartphone and upload it to the app, where they are then asked to fill out a questionnaire to assess their satisfaction with the meal and their physical condition.

[0405] 2. Data Analysis

[0406] The server analyzes the ingredients contained in the dish from the uploaded photo, and generates and presents advice using a generative AI model and prompts that instruct the server to generate advice regarding the presence or absence of allergens, estimated calories, and recommended drinks or foods to ingest based on the analysis results.

[0407] Example prompt sentence: After receiving the analysis results of the ingredients contained in the dish as input, "generate advice on the presence or absence of allergens, estimated calories, and recommended drinks or foods to ingest based on the analysis results of the ingredients."

[0408] "Since your last meal was high in calories, I would like to suggest a low-calorie meal next time."

[0409] 3. Meal suggestions

[0410] The server stores the user's diet history or questionnaire results in a database. The server analyzes the stored data, evaluates the user's dietary balance, and uses a generative AI model and a prompt to suggest the next meal.

[0411] Example prompt sentence: "User's food history data: 2023-10-01: Hamburgers, fries, high calories 2023-10-02: Salad, low calorie User survey results: 2023-10-01: Satisfaction: High, Health: Normal 2023-10-02: Satisfaction: Average, Health: Good Evaluate the user's dietary balance and suggest low-calorie menus for their next meal.

[0412] Sample meal suggestions for your next meal: "I suggest a salad and grilled chicken for your next meal."

[0413] 4. User Interface

[0414] The suggested menu is displayed on a smartphone or smart glasses.

[0415] In this way, a system can be built that makes suggestions for the next meal based on the user's eating history and survey results.

[0416] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0417] Step 1:

[0418] The user takes a photo of the meal using a smartphone or smart glasses.

[0419] Input: Food photo

[0420] Output: Captured photo data

[0421] Specific operation: The user takes a photo of the meal and uploads it to the server through the application.

[0422] Step 2:

[0423] The server receives the photograph and uses image analysis software (e.g., OpenCV) to identify the ingredients contained in the dish.

[0424] Input: Photographed photo data

[0425] Output: Analyzed material data

[0426] What it does: The server analyzes the photo data and applies image recognition algorithms to identify ingredients in the image.

[0427] Step 3:

[0428] Based on the analysis results, the server estimates the presence or absence of allergens and calories, and generates appropriate food and drink advice.

[0429] Input: Analyzed material data

[0430] Output: Presence or absence of allergens, estimated calories, food and drink advice

[0431] Specific operation: The server references nutritional information stored in the database and compares it with the analysis results to estimate the presence or absence of allergens and calories, and also generates appropriate food and drink recommendations.

[0432] Step 4:

[0433] The server stores the user's meal history and survey results in a database (e.g., SQLite).

[0434] Input: User's dietary history, survey results

[0435] Output: Saved data

[0436] Specific operation: The server receives the meal history and survey results entered by the user and stores them in a database.

[0437] Step 5:

[0438] The server analyzes the stored data and evaluates the user's dietary balance.

[0439] Input: Saved data

[0440] Output: Dietary balance evaluation results

[0441] Specific operation: The server uses a data analysis library (e.g., Pandas) to analyze the stored data and evaluate the user's dietary balance.

[0442] Step 6:

[0443] The server will make suggestions for the next meal based on the analysis results.

[0444] Input: Dietary balance assessment results

[0445] Output: Next meal suggestion

[0446] Specific operation: The server uses a generative AI model (e.g., TensorFlow) to generate the optimal meal menu for the user.

[0447] Step 7:

[0448] The suggested meal menu is displayed on the user's device (smartphone or smart glasses).

[0449] Input: Next meal suggestion

[0450] Output: Displayed meal menu

[0451] Specific operation: The server sends the proposed menu to the user's terminal and displays it visually through the application.

[0452] In addition, the emotion engine described below may be used to recognize the user's emotions, and advice may be generated and presented using a prompt sentence that instructs the system to generate advice based on the analysis results and the user's emotions, and a generative AI model.

[0453] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0454] "Example 1"

[0455] One embodiment of the present invention is a system that combines an emotion engine that recognizes the user's emotions. When a user takes a photo of a meal, this system analyzes the ingredients contained in the dish from the photo. Based on the analysis results, the system sends the user advice on whether or not there are any allergies, estimated calories, and recommended drinks and foods to accompany the meal. Furthermore, the emotion engine reads emotions from the user's facial expressions and tone of voice and makes meal suggestions that match those emotions. For example, if a user shows a happy expression, the emotion engine recognizes that joy and suggests meals that the user is likely to like.

[0456] "Example 2"

[0457] Another embodiment is a system that utilizes an emotion engine even when dining with multiple people. This system conducts a survey based on the participants' preferences and suggests dishes based on the survey results. Furthermore, the emotion engine reads the emotions of all participants and suggests meals that match those emotions. For example, if all participants look happy, the emotion engine will recognize that happiness and suggest meals that fit the happy atmosphere.

[0458] "Example 3"

[0459] Another example is a system that stores analysis results and survey results and uses them to suggest future meals. This system uses an emotion engine to read the user's emotions and suggests meals that match those emotions. For example, if the user showed a happy expression the last time they ate a particular dish, that information is saved and used as a reference when suggesting the next meal.

[0460] The processing flow of each embodiment will be described below.

[0461] "Example 1"

[0462] Step 1: The user takes a photo of their meal.

[0463] Step 2: The system analyzes the ingredients in the dish from the photo.

[0464] Step 3: Based on the analysis results, the user is informed of any allergies, estimated calories, and recommended drinks and foods to ingest.

[0465] Step 4: The emotion engine reads the user's emotions from their facial expressions and tone of voice.

[0466] Step 5: The emotion engine makes meal suggestions that match the emotions it has read.

[0467] "Example 2"

[0468] Step 1: The system surveys participants based on their preferences.

[0469] Step 2: Suggest recommended dishes based on the survey results.

[0470] Step 3: The emotion engine reads the emotions of all participants.

[0471] Step 4: The emotion engine makes meal suggestions that match the emotions it has read.

[0472] "Example 3"

[0473] Step 1: The system saves the analysis results and survey results.

[0474] Step 2: Based on the saved results, suggestions for future meals are made.

[0475] Step 3: The emotion engine reads the user's emotions.

[0476] Step 4: The emotion engine makes meal suggestions that match the emotions it has read.

[0477] Example 1

[0478] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0479] With conventional dietary analysis systems, even if a user took a photo of their meal, it was difficult to accurately analyze the ingredients contained in the dish from that photo. Furthermore, it was not possible to provide information on the presence or absence of allergens and estimated calories based on the analysis results, or to make meal suggestions based on the user's emotions. Furthermore, when dining with multiple people, it was difficult to make suggestions based on the preferences of the participants.

[0480] The identification process by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes means for taking a photo of the meal, means for transmitting the taken photo to the server, means for analyzing the transmitted photo, means for generating, based on the analysis results, information on the presence or absence of allergens, estimated calories, and advice on recommended drinks and foods to ingest, means for transmitting the generated advice to the user, means for recognizing the user's emotions, and means for suggesting meals based on the recognized emotions. This makes it possible to accurately analyze ingredients from a photo of the meal taken by the user and provide information on the presence or absence of allergens and estimated calories, as well as to suggest meals based on the user's emotions. Furthermore, when dining with multiple people, suggestions can be made based on the preferences of the participants.

[0481] The "means for taking photos of meals" refers to a device with a camera function that allows the user to take photos of meals.

[0482] The "means for transmitting the photographed photo to the server" is a device equipped with a communication function for transmitting the photographed photo data to the server via the Internet.

[0483] The "means for analyzing the transmitted photographs" refers to software that uses image recognition technology to analyze the photographic data transmitted to the server and identify the type of food and its ingredients.

[0484] "Means for generating advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to ingest based on the analysis results" refers to software equipped with an algorithm for generating advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods based on the analyzed data.

[0485] The "means for transmitting the generated advice to the user" is a device having a communication function for transmitting the generated advice to the user's terminal.

[0486] The "means for recognizing user emotions" is software that uses emotion analysis technology to analyze the user's facial expressions and tone of voice and recognize their emotions.

[0487] The "means for making meal suggestions based on recognized emotions" is software having an algorithm for making appropriate meal suggestions based on the recognized emotions of the user.

[0488] The "means for conducting a survey based on the preferences of participants when dining with multiple people" is a device equipped with a survey function for collecting the preferences of participants when dining with multiple people.

[0489] The "means for suggesting recommended dishes based on survey results" is software equipped with an algorithm for suggesting dishes suitable for participants based on the collected survey results.

[0490] "Means for saving analysis results and survey results and making meal suggestions based on those results" refers to software with data storage and suggestion functions that saves analysis results and survey results in a database and can be used to make meal suggestions for the next time and onwards.

[0491] This invention is a system that allows users to take photos of their meals and analyzes the ingredients contained in the dishes from the photos. By combining image recognition technology and emotion analysis technology, this system can provide more personalized meal recommendations to users.

[0492] Hardware and software used

[0493] Hardware: Devices such as smartphones and tablets, and servers

[0494] Software: Image recognition technology (e.g., TensorFlow, PyTorch), emotion engine (e.g., Affectiva)

[0495] Specific operation of the system

[0496] 1. User takes a photo of their meal:

[0497] The user opens the camera app on their smartphone or tablet and takes a photo of a meal. For example, the user takes a photo of curry rice.

[0498] 2. The device sends the photo to the server:

[0499] The device sends the captured photo to the server via the Internet, specifically by uploading the photo data to the server using an HTTP POST request.

[0500] 3. The server analyzes the image:

[0501] The server then analyzes the received photos using image recognition technology. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to input the photos into a neural network model to identify the type of dish and its ingredients. For example, a photo of curry rice could be analyzed to identify ingredients such as chicken, potatoes, carrots, and onions.

[0502] 4. The server generates the analysis results:

[0503] Based on the analysis results, the server generates advice on whether or not there are any allergies, estimated calories, and recommended drinks and foods to ingest together. For example, it may determine that curry rice does not contain any allergens, has an estimated calorie count of 500 kcal, and that the recommended drink is green tea.

[0504] 5. The server sends the analysis results to the user:

[0505] The server sends the generated analysis results to the user's device. Specifically, it returns the analysis results as an HTTP response.

[0506] 6. Emotion engine recognizes user emotions:

[0507] The emotion engine analyzes facial expressions and tone of voice when a user takes a photo to recognize the user's emotions. For example, it uses Affectiva's SDK to determine whether the user is showing a happy expression.

[0508] 7. Servers make emotion-based meal suggestions:

[0509] The server then suggests meals that match the user's emotions based on the analysis results of the emotion engine. For example, if the user shows a happy expression, the server will suggest a dessert that the user would like (e.g., chocolate cake).

[0510] Specific examples

[0511] User photo: Curry rice photo

[0512] Analysis result: Curry rice, ingredients (chicken, potato, carrot, onion)

[0513] Advice: No allergens, estimated calorie intake of 500kcal, recommended drink: green tea

[0514] Emotion recognition: If the user shows a happy expression, suggest desserts they might like (e.g., chocolate cake)

[0515] Prompt Sentence Examples

[0516] "I took a photo of curry rice. I'd like you to analyze the ingredients, tell me if there are any allergies, the estimated calories, and what drinks and foods I should eat with it. I'd also like you to make meal suggestions based on the user's emotions."

[0517] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0518] Step 1:

[0519] The user takes a photo of their meal.

[0520] Input: A user opens the camera app on their smartphone or tablet and takes a photo of their food.

[0521] Specific operation: The user opens the camera app, takes a photo of curry rice, and then taps the "Start analysis" button in the app.

[0522] Output: Captured photo data (JPEG format).

[0523] Step 2:

[0524] The device sends the photo to the server.

[0525] Input: Captured photo data (JPEG format).

[0526] Specific operation: The device sends the captured photo to the server via the Internet. Specifically, it uploads the photo data to the server using an HTTP POST request. A progress bar is displayed during the transmission, and a "Transmission complete" message is displayed once the transmission is complete.

[0527] Output: The photo data sent to the server.

[0528] Step 3:

[0529] The server analyzes the image.

[0530] Input: Photo data sent to the server.

[0531] How it works: The server temporarily stores the received photo and then uses a deep learning framework such as TensorFlow or PyTorch to input the photo into a neural network model. The model analyzes the photo and identifies the type of dish and its ingredients. For example, a photo of curry rice could be analyzed to identify ingredients such as chicken, potatoes, carrots, and onions. The analysis takes a few seconds.

[0532] Output: Analysis results (type of food and ingredients).

[0533] Step 4:

[0534] The server generates the analysis results.

[0535] Input: Analysis results (type of food and ingredients).

[0536] Specific operation: Based on the analysis results, the server generates advice on whether or not there are any allergies, the estimated calories, and the recommended drinks and foods to ingest together. For example, it generates the information that curry rice does not contain any allergens, has an estimated calorie count of 500 kcal, and the recommended drink is green tea.

[0537] Output: Generated advice (presence of allergies, estimated calories, recommended drinks and foods).

[0538] Step 5:

[0539] The server sends the analysis results to the user.

[0540] Input: Generated advice (presence of allergies, estimated calories, recommended drinks and foods).

[0541] Specific operation: The server sends the generated analysis results to the user's device. Specifically, it returns the analysis results as an HTTP response.

[0542] Output: Analysis results sent to the user's device.

[0543] Step 6:

[0544] The emotion engine recognizes the user's emotions.

[0545] Input: The user's facial expression and tone of voice.

[0546] How it works: The emotion engine analyzes facial expressions and tone of voice when a user takes a photo to recognize the user's emotions. For example, it uses Affectiva's SDK to determine whether the user is showing a happy expression.

[0547] Output: Recognized emotion data (e.g., happiness, sadness, surprise, etc.).

[0548] Step 7:

[0549] Servers make emotion-based meal suggestions.

[0550] Input: Recognized emotion data.

[0551] Specific operation: The server makes meal suggestions that match the user's emotions based on the analysis results of the emotion engine. For example, if the user shows a happy expression, the server will suggest a dessert that the user would like (e.g., chocolate cake).

[0552] Output: Emotion-based meal suggestions.

[0553] (Application example 1)

[0554] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0555] Conventional dietary analysis systems allow users to take photos of their meals and analyze the ingredients contained in the dish, but they lack functionality such as suggesting meals based on the user's emotions, detecting allergies, and calculating estimated calories. Another problem is that they are unable to make suggestions based on the preferences of participants when dining with others. Furthermore, they lack the ability to save analysis and survey results and use them for future meal suggestions. To solve these problems, a system with more advanced analysis capabilities and a suggestion function based on the user's emotions is needed.

[0556] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0557] In this invention, the server includes a means for taking photos of meals, a means for analyzing the ingredients contained in the dishes from the photos, a means for sending information on the presence or absence of allergens, estimated calories, and recommendations on recommended drinks and other foods to accompany the meal based on the analysis results, a means for recognizing the user's emotions, a means for suggesting additional dishes and drinks based on the recognized emotions, a means for conducting a survey based on the preferences of the participants in the case of a meal with multiple people, a means for suggesting recommended dishes based on the survey results, and a means for saving the analysis results and the survey results and making meal suggestions based on these results for future meals. This enables meal suggestions based on the user's emotions, detection of allergens, and calculation of estimated calories, making it possible to make suggestions based on the preferences of the participants even when dining with multiple people. The analysis results and the survey results can also be saved and used for future meal suggestions.

[0558] "Means for taking photos of meals" refers to devices or functions that allow users to take photos of meals, and includes photography devices such as smartphones and cameras.

[0559] "Means for analyzing the ingredients contained in a dish from a photograph" refers to image recognition technology and algorithms for identifying the ingredients contained in a dish based on a photograph.

[0560] "A means of sending advice on whether or not an item contains allergens, estimated calories, and recommended drinks and foods to ingest based on the analysis results" is a function that calculates whether or not an item contains allergens and calories based on analyzed ingredient information, and provides users with advice on appropriate foods and drinks.

[0561] "Means of recognizing user emotions" refers to technologies and algorithms for reading emotions from a user's facial expressions, tone of voice, etc.

[0562] "Means for suggesting additional dishes and drinks based on recognized emotions" is a function for suggesting appropriate dishes and drinks according to the user's emotions.

[0563] "In the case of a meal with multiple people, a means of conducting a survey based on the preferences of the participants" refers to a survey function for collecting the preferences of each participant in a meal with multiple people.

[0564] "Means for suggesting recommended dishes based on survey results" is a function for suggesting dishes suitable for all participants based on the collected survey results.

[0565] "Means for saving analysis results and survey results and making meal suggestions for the next time and beyond based on those results" refers to a function that saves past analysis results and survey results in a database and makes meal suggestions for the next time and beyond based on that data.

[0566] A system for implementing this invention has the ability to allow a user to take a photo of a meal and analyze the ingredients contained in the dish from the photo. The analysis is performed using image recognition technology, specifically a neural network based on deep learning. Based on the analysis results, the system sends the user information such as whether or not there are any allergies, estimated calories, and recommendations on recommended drinks and foods to accompany the meal. The system also has the ability to recognize the user's emotions and suggest additional dishes and drinks based on those emotions.

[0567] Hardware and software used

[0568] Hardware: Smartphones, cameras

[0569] Software: Python, Keras, OpenCV, EmotionRecognizer library

[0570] Data processing and calculation

[0571] Image Recognition

[0572] Users take a photo of their meal with their smartphone camera and upload it to the app, where the server analyzes the photo using Keras image recognition models to identify the ingredients in the dish.

[0573] Allergen detection

[0574] The server detects whether or not an allergen is present based on the analysis results, and notifies the user if an allergy is present.

[0575] Calorie Counting

[0576] The server calculates the estimated calories from the ingredients of the dish and displays them to the user.

[0577] emotion recognition

[0578] The EmotionRecognizer library is used to read the user's emotions from their facial expressions and tone of voice, and the server then suggests appropriate dishes and drinks based on the user's emotions.

[0579] Dining with multiple people

[0580] When dining with multiple people, the server will conduct a survey based on the participants' preferences and then recommend dishes based on the results.

[0581] Data storage and next proposal

[0582] The server stores the analysis results and survey results in a database and uses this data to make meal suggestions for the next time and beyond.

[0583] Specific examples

[0584] When a user uploads a photo of a pizza they have taken with their smartphone to the app, the server analyzes the ingredients (cheese, tomato, pepperoni, etc.) contained in the pizza and displays the presence of allergens and estimated calories. Furthermore, if the user shows a happy expression, the server will suggest, "Would you like dessert?"

[0585] Prompt Sentence Examples

[0586] Analyze photos of food taken by users to identify the ingredients contained in them, calculate allergens and estimated calories and notify the user, and recognize the user's emotions and suggest additional dishes or drinks that match those emotions.

[0587] The above is an embodiment of the present invention.

[0588] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0589] Step 1:

[0590] Users take photos of their meals using their smartphone camera and upload them to the app.

[0591] Input: Food photo

[0592] Output: Uploaded photo data

[0593] Specific operation: A user takes a photo of a meal using the smartphone camera and uploads the photo to the application, which then sends the photo data to the server.

[0594] Step 2:

[0595] The server receives the uploaded photo and uses image recognition models to analyze the ingredients contained in the dish.

[0596] Input: Uploaded photo data

[0597] Output: Analyzed material information

[0598] Specific operation: The server uses Keras image recognition model to analyze the photo data and identify the ingredients contained in the dish. As a result of the analysis, it generates ingredient information.

[0599] Step 3:

[0600] The server detects whether or not an allergen is present based on the analysis results and notifies the user.

[0601] Input: Analyzed material information

[0602] Output: Notification of allergen presence / absence

[0603] Specific operation: The server detects whether the food contains an allergen based on the analyzed ingredient information. If so, it sends a notification to the user.

[0604] Step 4:

[0605] The server calculates the estimated calories based on the analysis results and displays them to the user.

[0606] Input: Analyzed material information

[0607] Output: Estimated calories displayed

[0608] Specific operation: The server calculates the calories of each ingredient based on the ingredient information, calculates the total calories, and displays the calculated calorie information to the user.

[0609] Step 5:

[0610] The server recognizes the user's emotions from their facial expressions and tone of voice.

[0611] Input: User facial expressions and tone of voice

[0612] Output: Recognized emotion information

[0613] Specific operation: The server uses the EmotionRecognizer library to analyze the user's facial expressions and tone of voice to recognize emotions, and generates recognized emotion information.

[0614] Step 6:

[0615] Servers suggest additional dishes and drinks based on perceived emotions.

[0616] Input: Recognized emotion information

[0617] Output: Additional food and drink suggestions

[0618] Specific operation: Based on the recognized emotion information, the server suggests dishes and drinks that suit the user's emotion and notifies the user of the suggestions.

[0619] Step 7:

[0620] The server stores the analysis results and survey results in a database and uses them to make meal suggestions for future visits.

[0621] Input: Analysis results, survey results

[0622] Output: Saved data, suggestions for future use

[0623] Specific operation: The server stores the analysis results and survey results in a database, and uses this data to make meal suggestions for the next time and beyond. The saved data is used to make suggestions that are appropriate for the user.

[0624] Example 2

[0625] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0626] Conventional meal recommendation systems can take into account participants' preferences and allergy information, but they have the problem of not being able to make suggestions that reflect participants' emotions or real-time situations. It is also difficult to determine whether the suggested dishes match the preferences of all participants, making it difficult to increase meal satisfaction. Furthermore, the system lacked the functionality to utilize past data to make suggestions for future meals, making it difficult to continuously improve satisfaction.

[0627] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0628] In this invention, the server includes a means for taking photos of meals, a means for analyzing the ingredients contained in the dishes from the photos, a means for sending information on the presence or absence of allergens, estimated calories, and advice on recommended drinks and foods to accompany the dishes based on the analysis results, a means for conducting a survey based on participants' preferences, a means for suggesting recommended dishes based on the survey results, a means for reading participants' emotions and suggesting meals that match those emotions, and a means for inputting prompts into a generative AI model to generate optimal dish suggestions. This enables real-time meal suggestions that take participants' preferences and emotions into account, thereby increasing meal satisfaction. Furthermore, by utilizing past data to make suggestions for future meals, continuous improvement in satisfaction is possible.

[0629] "Means for taking photos of food" refers to devices or functions for taking photos of food. Specifically, this applies to cameras and the camera functions of smartphones.

[0630] "Means for analyzing ingredients in a dish" refers to devices and software that identify and analyze ingredients in a dish from a photograph. Specifically, image recognition technology and ingredient databases are used.

[0631] "Means of sending advice on whether or not there are allergens, estimated calories, and recommended drinks and foods to ingest" refers to devices and software that provide information on whether or not there are allergens and calories based on the analysis results, and give advice on appropriate foods and drinks.

[0632] "Means for surveying participants based on their preferences" refers to devices or software used to collect participants' dietary preferences or restrictions, such as survey forms or applications.

[0633] "Means for suggesting recommended dishes based on survey results" refers to devices or software that analyze the collected survey results and suggest the best dishes for participants.

[0634] "Means to read participants' emotions and suggest meals that match those emotions" refers to devices or software that analyze emotions from participants' facial expressions, voices, etc., and suggest appropriate meals based on those emotions. Specifically, this applies to emotion engines and facial expression recognition technology.

[0635] "Means for inputting prompt sentences into a generative AI model and generating optimal dish suggestions" refers to devices or software that input prompt sentences into a generative AI model and generate optimal dish suggestions as a result. Specifically, it uses natural language processing technology.

[0636] "Means for saving analysis results and survey results and making meal suggestions for the next time and beyond based on those results" refers to devices and software that save analysis results and survey results in a database and make meal suggestions for the next time and beyond based on that data.

[0637] This invention is a system that suggests the best dishes for a group meal based on the preferences and emotions of the participants. The system uses a questionnaire and an emotion engine to analyze the preferences and emotions of the participants and suggests dishes based on that.

[0638] Hardware and software used

[0639] Hardware: Servers, devices (smartphones, tablets, PCs)

[0640] Software: Survey interface, sentiment engine, database, generative AI model

[0641] Specific operation of the system

[0642] 1. Conducting a survey

[0643] The user uses a terminal to access the questionnaire interface provided by the system.

[0644] The user inputs their preferences and dietary restrictions into the questionnaire interface, such as "I like fish dishes," "I'm a vegetarian," "I like spicy food," "I don't have allergies," and "I like Japanese food."

[0645] The terminal transmits the inputted questionnaire data to the server in real time.

[0646] 2. Analysis of the survey data

[0647] The server stores the received questionnaire data in a database.

[0648] The server analyzes the survey data stored in the database and compiles the preferences of all participants. For example, if it turns out that everyone likes fish, it uses that information to move on to the next step.

[0649] The server creates a prompt to be input to the generative AI model based on the analysis results. For example, it generates a prompt such as "fish dishes, vegetarian, spicy dishes, no allergies, Japanese food."

[0650] 3. Food suggestions

[0651] The server inputs a prompt sentence into the generative AI model.

[0652] The generative AI model generates optimal dish suggestions based on the prompt, for example, "a Japanese-style fish dish that is not spicy."

[0653] The server transmits the generated cooking suggestions to the terminal.

[0654] The user checks the suggested dishes through the terminal. For example, "a mild Japanese-style fish dish" is displayed on the terminal screen.

[0655] 4. Use of Emotion Engine (Alternative Embodiment)

[0656] The user sends their facial expressions to the system using the device's camera, for example, showing a happy expression while eating.

[0657] The terminal transmits the facial expression data to the server in real time.

[0658] The server uses an emotion engine to analyze the facial expression data and interpret the emotions of all participants. For example, it recognizes that everyone is looking happy.

[0659] The server creates a prompt sentence to be input to the generative AI model based on the emotion, for example, "A dish that matches a fun atmosphere."

[0660] The generative AI model generates emotion-matching food suggestions based on the prompt, for example, "Party Platter."

[0661] The server transmits the generated cooking suggestions to the terminal.

[0662] The user checks the proposed dishes through the terminal. For example, "Party Platter" is displayed on the terminal screen.

[0663] Specific examples

[0664] For example, if there are five participants, the system works as follows:

[0665] 1. Conducting a survey

[0666] User A enters "I like fish dishes."

[0667] User B enters "vegetarian."

[0668] User C enters "I like spicy food."

[0669] User D enters "No allergies."

[0670] User E enters "I like Japanese food."

[0671] 2. Analysis of questionnaire data

[0672] The server aggregates this data and generates prompts to suggest dishes that will satisfy everyone.

[0673] 3. Food suggestions

[0674] The generative AI model is given the prompt sentence "Fish dish, vegetarian, spicy dish, no allergies, Japanese food."

[0675] The generative AI model suggests "non-spicy Japanese-style fish dishes."

[0676] The server sends this suggestion to the terminal, and the user confirms the suggested dish.

[0677] 4. Utilizing the Emotion Engine

[0678] If the user shows a happy expression, the emotion engine will recognize that happiness.

[0679] The generative AI model is given the prompt, "Food that matches a fun atmosphere."

[0680] The generative AI model suggests "a platter of food for a party."

[0681] The server sends this suggestion to the terminal, and the user confirms the suggested dish.

[0682] The above is a specific embodiment of this system.

[0683] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0684] Step 1:

[0685] Conducting a survey

[0686] The user uses a terminal to access the questionnaire interface provided by the system.

[0687] The user inputs their preferences and dietary restrictions into the questionnaire interface, such as "I like fish dishes," "I'm a vegetarian," "I like spicy food," "I don't have allergies," and "I like Japanese food."

[0688] Input: User preferences and dietary restrictions

[0689] The terminal transmits the inputted questionnaire data to the server in real time.

[0690] Output: Survey data sent to the server

[0691] Step 2:

[0692] Analysis of survey data

[0693] The server stores the received questionnaire data in a database.

[0694] The server analyzes the survey data stored in the database and compiles the preferences of all participants. For example, if it turns out that everyone likes fish, it uses that information to move on to the next step.

[0695] Input: Survey data stored in a database

[0696] The server creates a prompt to be input to the generative AI model based on the analysis results. For example, it generates a prompt such as "fish dishes, vegetarian, spicy dishes, no allergies, Japanese food."

[0697] Output: Prompt sentence to be input to the generative AI model

[0698] Step 3:

[0699] Cooking suggestions

[0700] The server inputs a prompt sentence into the generative AI model.

[0701] The generative AI model generates optimal dish suggestions based on the prompt, for example, "a Japanese-style fish dish that is not spicy."

[0702] Input: The prompt given to the generative AI model

[0703] The server transmits the generated cooking suggestions to the terminal.

[0704] Output: Food suggestions sent to device

[0705] The user checks the suggested dishes through the terminal. For example, "a mild Japanese-style fish dish" is displayed on the terminal screen.

[0706] Step 4:

[0707] Use of Emotion Engine (another embodiment)

[0708] The user sends their facial expressions to the system using the device's camera, for example, showing a happy expression while eating.

[0709] Input: User's facial expression data

[0710] The terminal transmits the facial expression data to the server in real time.

[0711] The server uses an emotion engine to analyze the facial expression data and interpret the emotions of all participants. For example, it recognizes that everyone is looking happy.

[0712] Input: facial expression data

[0713] The server creates a prompt sentence to be input to the generative AI model based on the emotion, for example, "A dish that matches a fun atmosphere."

[0714] Output: Prompt sentence to be input to the generative AI model

[0715] The generative AI model generates emotion-matching food suggestions based on the prompt, for example, "Party Platter."

[0716] Input: The prompt given to the generative AI model

[0717] The server transmits the generated cooking suggestions to the terminal.

[0718] Output: Food suggestions sent to device

[0719] The user checks the proposed dishes through the terminal. For example, "Party Platter" is displayed on the terminal screen.

[0720] (Application example 2)

[0721] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0722] Conventional meal recommendation systems make suggestions based on participants' preferences and dietary restrictions, but they have the problem of not being able to make suggestions that are tailored to the participants' emotions or the atmosphere of the occasion. Furthermore, when dining with multiple people, it is difficult to suggest dishes that will satisfy everyone. Furthermore, there was a need for a system that can easily make these suggestions using devices such as smartphones or smart glasses.

[0723] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0724] In this invention, the server includes means for acquiring images of meals, means for analyzing the ingredients contained in the dishes from the acquired images, means for providing the presence or absence of allergens, estimated calories, and advice on recommended drinks and foods to ingest based on the analysis results, means for conducting a questionnaire based on the preferences of participants in the case of a meal with multiple people, means for suggesting recommended dishes based on the questionnaire results, and means for reading the emotions of participants using an emotion engine and suggesting optimal dishes based on those emotions. This makes it possible to suggest optimal dishes based on the preferences and emotions of all participants.

[0725] The "means for acquiring an image of a meal" refers to a device or function for taking an image of a meal using a camera or the camera function of a smartphone and acquiring the image data.

[0726] "Means for analyzing the ingredients contained in a dish from an acquired image" refers to a device or function that uses image analysis technology to identify the ingredients contained in a dish from a photographed image of the food and extract that information.

[0727] "Means for providing advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to be consumed together based on the analysis results" refers to a device or function that calculates the presence or absence of allergens and estimated calories based on the results of ingredient analysis, and also provides advice on recommended drinks and foods to be consumed together.

[0728] "Means for conducting a questionnaire based on the preferences of participants when dining with multiple people" refers to a device or function for collecting the dietary preferences and restrictions of multiple participants in the form of a questionnaire.

[0729] The "means for proposing recommended dishes based on survey results" refers to a device or function for proposing dishes that will satisfy all participants based on the collected survey results.

[0730] "Means for reading participants' emotions using an emotion engine and suggesting the most suitable dish based on those emotions" refers to a device or function that uses emotion recognition technology to read emotions from participants' facial expressions and tone of voice, and suggests the most suitable dish based on those emotions.

[0731] "Means for saving analysis results and survey results and making meal suggestions for the next time and beyond based on those results" refers to a device or function that saves past analysis results and survey results in a database and makes meal suggestions for the next time and beyond based on that data.

[0732] "Applications installed on smartphones, smart glasses, head-mounted displays, or robots" are software applications that are installed on these devices and designed to be easily accessible to users.

[0733] This invention is a system that suggests optimal dishes for a group meal based on the preferences and emotions of the participants. This system is implemented as an application installed on a smartphone, smart glasses, a head-mounted display, or a robot.

[0734] System configuration

[0735] The system consists of the following main components:

[0736] 1. Method for acquiring food images: Take photos of food using the camera function of a smartphone or smart glasses.

[0737] 2. A method for analyzing the ingredients contained in the food from the captured image: Using image analysis technology, the ingredients contained in the food are identified from the captured image. Specifically, an image processing library such as OpenCV is used.

[0738] 3. A means of providing advice on the presence or absence of allergens, estimated calories, and recommended beverages and foods to consume together based on the analysis results: Based on the results of the ingredient analysis, the presence or absence of allergens and estimated calories are calculated, and advice is provided on the recommended beverages and foods to consume together.

[0739] 4. For group meals, a survey based on participants' preferences is conducted: A survey is conducted to collect information on each participant's dietary preferences and restrictions. The survey is conducted through a smartphone application interface.

[0740] 5. A method for suggesting recommended dishes based on survey results: Based on the collected survey results, dishes that will satisfy all participants are suggested.

[0741] 6. Using an emotion engine to read participants' emotions and suggest the best dish based on those emotions: Using emotion recognition technology, we read participants' emotions from their facial expressions and tone of voice, and suggest the best dish based on those emotions. Specifically, we use the EmotionRecognizer library.

[0742] 7. A means of saving analysis results and survey results and making meal suggestions for the next time and beyond based on those results: Past analysis results and survey results are saved in a database, and meal suggestions for the next time and beyond are made based on that data.

[0743] Processing flow

[0744] 1. The user takes a picture of the meal using the smartphone camera.

[0745] 2. The device analyzes the captured image and identifies the ingredients in the dish.

[0746] 3. Based on the analysis results, the device will calculate the presence or absence of allergens and estimated calories, and also provide advice on recommended beverages and foods to consume.

[0747] 4. If the user is dining with multiple people, collect the preferences and dietary restrictions of all participants through a survey.

[0748] 5. The device will suggest recommended dishes based on the survey results.

[0749] 6. The device uses an emotion engine to read the participant's emotions and suggests the best dish based on those emotions.

[0750] 7. The server stores the analysis results and survey results and uses them to make meal suggestions for future visits.

[0751] Specific examples

[0752] For example, at a family dinner, the app can suggest dishes that suit everyone's tastes and mood that day, or at a party with friends, it can suggest dishes that everyone will enjoy.

[0753] Prompt Sentence Examples

[0754] "If you're having a family dinner, suggest dishes that fit everyone's tastes and their mood that day. For example, if everyone likes fish and looks happy, what dishes would you suggest?"

[0755] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0756] Step 1:

[0757] The user takes a picture of the meal using the smartphone camera.

[0758] Input: Food image

[0759] Output: Captured image data

[0760] Specific operation: The user launches the camera app on their smartphone and takes a picture of their meal. The captured image is saved in the application.

[0761] Step 2:

[0762] The device analyzes the captured image and identifies the ingredients contained in the dish.

[0763] Input: Photographed image data

[0764] Output: Information on ingredients contained in the dish

[0765] Specific operation: The device uses image processing libraries such as OpenCV to analyze the ingredients of the food from the captured image. As a result of the analysis, the main ingredients contained in the food are identified.

[0766] Step 3:

[0767] Based on the analysis results, the device calculates the presence or absence of allergens and estimated calories, and also provides advice on recommended beverages and foods to consume.

[0768] Input: Information about ingredients contained in the dish

[0769] Output: Presence or absence of allergens, estimated calories, and recommended drinks and foods

[0770] How it works: The device checks for allergens based on ingredient information, calculates estimated calories, and generates recommendations for drinks and foods to ingest based on the ingredients.

[0771] Step 4:

[0772] If a user is dining with multiple people, the preferences and dietary restrictions of all participants are collected through a questionnaire.

[0773] Input: Survey responses regarding participants' preferences and dietary restrictions

[0774] Output: Survey result data

[0775] Specific operation: The user uses the survey function of the smartphone application to enter the preferences and dietary restrictions of all participants. The survey results are saved within the application.

[0776] Step 5:

[0777] The device will suggest recommended dishes based on the survey results.

[0778] Input: Survey result data

[0779] Output: Recommended dish list

[0780] Specific operation: The device analyzes the survey results and suggests dishes that will satisfy all participants. The suggested dishes are then displayed to the user.

[0781] Step 6:

[0782] The device uses an emotion engine to read the participant's emotions and suggests the best dish based on those emotions.

[0783] Input: Participant's facial expressions and tone of voice

[0784] Output: Emotion data, best recipe list

[0785] How it works: The device uses the EmotionRecognizer library to analyze participants' facial expressions and tone of voice through the camera and microphone. Based on the emotional data, it suggests the best dishes to serve.

[0786] Step 7:

[0787] The server stores the analysis results and survey results and uses them to make meal suggestions for future visits.

[0788] Input: Analysis result data, questionnaire result data

[0789] Output: Saved data, meal suggestions for the next time

[0790] Specific operation: The server stores the analysis results and survey results in a database and accumulates data to be used for future meal suggestions.

[0791] Example 3

[0792] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0793] Conventional meal recommendation systems do not take into account the user's eating history or emotional state when making suggestions, and therefore do not contribute sufficiently to improving the user's health management and satisfaction. Furthermore, when dining with multiple people, it is difficult to make suggestions that take into account the preferences and allergy information of all participants.

[0794] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0795] In this invention, the server includes: a means for taking photos of meals; a means for analyzing the ingredients contained in the meals from the photos; a means for sending, based on the analysis results, information on the presence or absence of allergens, estimated calories, and advice on recommended accompanying foods and drinks; a means for inputting the user's meal data and questionnaire results; a means for saving data; a means for analyzing the saved data; a means for generating next meal suggestions based on the analysis results; a means for displaying the generated meal suggestions to the user; and a means for analyzing the user's emotions using an emotion analysis engine and making meal suggestions based on the results. This enables meal suggestions to be made taking into account the user's meal history and emotional state, contributing to the user's health management and improved satisfaction. Furthermore, even for meals with multiple people, suggestions can be made taking into account the preferences and allergy information of all participants.

[0796] "Means for taking photos of food" refers to devices or functions that allow users to take photos of food. Specifically, this applies to smartphones and digital cameras.

[0797] "Means for analyzing ingredients contained in a dish from a photograph" refers to technologies and algorithms that analyze photographs of food and identify the ingredients contained therein, such as image analysis software and machine learning models.

[0798] "A means of sending advice on whether or not there are any allergens, estimated calories, and recommended drinks and foods to ingest" refers to a function that provides users with information on whether or not there are any allergens, calorie information, and suggestions for appropriate drinks and foods based on the analysis results.

[0799] "Means for inputting user dietary data and survey results" refers to interfaces or devices that allow users to input their dietary details and survey results. Specifically, this applies to smartphone apps and web forms.

[0800] "Means for storing data" refers to a database or storage system for storing the meal data and survey results entered by users.

[0801] "Means for analyzing stored data" refers to technologies and algorithms for analyzing a user's dietary history and health status based on stored data, such as data analysis software and machine learning models.

[0802] "Means for generating next meal suggestions based on analysis results" refers to a function for making next meal suggestions to the user based on the analysis results. Specifically, this applies to a suggestion algorithm or recommendation engine.

[0803] The "means for displaying the generated meal suggestions to the user" refers to an interface or device for displaying the generated meal suggestions to the user, such as a smartphone app or a web page.

[0804] "Means of analyzing user emotions using an emotion analysis engine and making meal suggestions based on the results" refers to the technology and algorithms used to analyze user emotions and the functionality used to make meal suggestions based on the results. Emotion analysis software and machine learning models fall into this category.

[0805] MODE FOR CARRYING OUT THE INVENTION

[0806] The present invention is a system for making meal suggestions that take into account a user's eating history and emotional state. Specific embodiments of this system will be described below.

[0807] System configuration

[0808] The system consists of the following main components:

[0809] 1. Terminal: A device that users use to input their dietary data and survey results. Specifically, this applies to smartphones and PCs.

[0810] 2. Server: The central component responsible for data storage, analysis, and recommendation generation. It includes a database server and a sentiment analysis engine.

[0811] 3. Database: A storage system for saving user dietary data and survey results. Specifically, this is a database management system such as MySQL.

[0812] 4. Emotion analysis engine: Software for analyzing user emotions. Specifically, Microsoft® Azure® Emotion API is one such example.

[0813] Program processing

[0814] The server receives the meal data and questionnaire results entered by the user from their device and stores them in a database. The stored data is referenced when making the next meal suggestion. The server retrieves the user's meal history from the database and analyzes it. Based on the analysis results, it generates a next meal suggestion and sends it to the user's device.

[0815] Furthermore, a sentiment analysis engine is used to analyze the user's emotions. For example, if the user shows a happy expression after eating a particular dish, this information is saved and used as a reference when suggesting a meal next time.

[0816] Specific examples

[0817] As a concrete example, let's say a user had "pizza" for their last meal. If the user's calorie intake was high at that time, the next time the system suggests a meal, it will suggest "salad" or "low-calorie soup." Also, if the user shows a happy expression after eating pizza, that information will be saved and the next time it suggests "pizza" or a "similar dish."

[0818] Prompt Sentence Examples

[0819] An example of a prompt to input to a generative AI model is as follows:

[0820] Generate a program that will suggest the next meal based on the user's eating history and the results of emotion analysis. If the user last ate a high-calorie meal, suggest a low-calorie meal, and if the user showed a happy expression after eating a particular dish, suggest that dish again.

[0821] In this way, the server realizes a system that makes meal suggestions for the next time and thereafter based on the user's meal history and emotion analysis results. The flow of the specification process in the third embodiment will be described with reference to FIG.

[0822] Program processing flow

[0823] Step 1: Entering user's dietary data and survey results

[0824] After finishing a meal, users use a device (smartphone or PC) to input information such as the meal contents, calories, and satisfaction level. They also enter data such as their impressions and physical condition in a questionnaire format.

[0825] Input: Meal details, calories, satisfaction, impressions, physical condition

[0826] Output: Sending input data

[0827] Specific behavior:

[0828] The user opens the dedicated application and inputs the meal details.

[0829] The user answers a questionnaire and inputs their level of satisfaction and physical condition.

[0830] The terminal transmits the input data to the server.

[0831] Step 2: Server saves data

[0832] The server stores the meal data and questionnaire results entered by the user in a database. The stored data is organized for each user and is referenced the next time meal suggestions are made.

[0833] Input: Meal data and survey results submitted by the user

[0834] Output: Save to database

[0835] Specific behavior:

[0836] The server receives the data sent by the user.

[0837] The server stores the received data in a database.

[0838] Step 3: Parse the data on the server

[0839] The server analyzes the stored data and evaluates the user's dietary balance and emotional state. For example, if the user eats a high-calorie meal, the server records that information and reflects it in the next meal recommendation.

[0840] Input: Dietary data stored in the database and questionnaire results

[0841] Output: Analysis results (calorie intake, satisfaction rating, etc.)

[0842] Specific behavior:

[0843] The server retrieves the user's meal history from the database.

[0844] The server analyzes the acquired data and evaluates calorie intake and satisfaction.

[0845] Step 4: Server generates next meal suggestions

[0846] The server generates meal suggestions based on the analysis results. For example, if a user ate a high-calorie meal last time, it will suggest a low-calorie meal. It also uses an emotion engine to suggest dishes that made the user happy.

[0847] Input: Analysis results, emotion analysis results

[0848] Output: Next meal suggestion

[0849] Specific behavior:

[0850] The server generates meal suggestions based on the analysis results.

[0851] The server uses an emotion engine to analyze the user's emotion data and reflects it in the proposal.

[0852] Step 5: Displaying meal suggestions to the user

[0853] The server sends the generated meal suggestions to the user's terminal, where the user can check the next meal suggestions.

[0854] Input: Next meal suggestion

[0855] Output: Proposal displayed on user's device

[0856] Specific behavior:

[0857] The server transmits the generated meal suggestions to the user's terminal.

[0858] The user checks the suggested meals on the terminal.

[0859] In this way, the server realizes a system that makes meal suggestions for the next time and beyond based on the user's meal history and emotion analysis results.

[0860] (Application example 3)

[0861] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0862] Conventional meal recommendation systems often make suggestions based on the user's eating history and survey results, but they lack suggestions that take into account the user's emotions and past eating history. Furthermore, when dining with multiple people, it is difficult to make suggestions that reflect the preferences of all participants. Furthermore, there is a need for a system that can easily make these suggestions using devices such as smartphones or smart glasses.

[0863] The identification process performed by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a means for taking a photo of the meal, a means for analyzing the ingredients contained in the dish from the taken photo, a means for sending information on the presence or absence of allergens, estimated calories, and advice on recommended drinks and foods to accompany the meal based on the analysis results, a means for saving the analysis results and questionnaire results and making meal suggestions based on those results, and a means for reading the user's emotions using an emotion engine and making meal suggestions tailored to those emotions. This enables meal suggestions that take into account the user's emotions and past eating history, and even in cases of multiple people, suggestions that reflect the preferences of all participants. Furthermore, a system can be realized that allows these suggestions to be easily accessed using devices such as smartphones and smart glasses.

[0864] "Means for taking photos of meals" refers to camera functions or applications that allow users to take photos of their meals.

[0865] "Means for analyzing ingredients contained in a dish from a photograph" refers to image analysis technology that analyzes photographs of food and identifies the ingredients and components contained therein.

[0866] "A means of sending advice on whether or not there are any allergens, estimated calories, and recommended drinks and foods to ingest based on the analysis results" refers to a system that uses analyzed dietary information to determine whether or not there are any allergens, calculate calories, and even suggest appropriate drinks and foods.

[0867] "A means of saving analysis results and survey results, and making meal suggestions for the next time and beyond based on those results" refers to a system that saves a user's meal history and survey results in a database and makes meal suggestions for the next time based on those results.

[0868] "A means of reading a user's emotions using an emotion engine and suggesting meals that match those emotions" refers to technology that analyzes emotions from a user's facial expressions and behavior and suggests appropriate meals based on those emotions.

[0869] "A means of conducting a survey based on participants' preferences when dining with multiple people" refers to a system that collects information on each user's preferences and allergies in the form of a survey when multiple users are dining together.

[0870] "Means for suggesting recommended dishes based on survey results" refers to a system for suggesting dishes suitable for all participants based on the collected survey results.

[0871] "Application installed on a smartphone, smart glasses, head-mounted display, or robot" refers to software that is installed on a device used by a user and that runs the meal recommendation system.

[0872] The following system configuration will be described as an embodiment of the present invention.

[0873] System Configuration

[0874] The system can be used by users using smartphones, smart glasses, head-mounted displays, or robots. The system consists of the following main components:

[0875] 1. A way to take photos of your food

[0876] Users take photos of their meals using the camera function on their smartphone or smart glasses.

[0877] 2. A method for analyzing ingredients contained in a dish from a photograph

[0878] The server uses image analysis software to analyze the photos and identify the ingredients in the dish, using image recognition technology and machine learning models.

[0879] 3. A method to send advice on whether or not you have any allergies, estimated calories, and recommended drinks and foods to ingest based on the analysis results

[0880] Based on the analysis results, the server calculates whether or not there are any allergens and the estimated calories, and sends the user advice on appropriate drinks and foods.

[0881] 4. A means to save analysis results and survey results and make meal suggestions based on those results for future meals.

[0882] The server stores the user's meal history and survey results in a database and refers to this data when making the next meal suggestion.

[0883] 5. A method to read the user's emotions using an emotion engine and suggest meals that match those emotions.

[0884] The server uses an emotion engine to analyze the user's emotions from their facial expressions and behavior, and then suggests appropriate meals based on those emotions.

[0885] 6. For group meals, survey participants based on their preferences.

[0886] When dining with multiple people, the server collects preferences and allergy information from all participants in a questionnaire format.

[0887] 7. A way to suggest dishes based on survey results

[0888] Based on the collected survey results, the server will suggest dishes suitable for all participants.

[0889] Processing flow

[0890] 1. Users take photos of their meals

[0891] Users take photos of their food using their smartphone or smart glasses.

[0892] 2. Photo Analysis

[0893] The server uses image analysis software to analyze the photos and identify the ingredients contained in the dish.

[0894] 3. Submitting Advice

[0895] Based on the analysis results, the server calculates whether or not there are any allergens and the estimated calories, and sends the user advice on appropriate drinks and foods.

[0896] 4. Data storage

[0897] The server stores the analysis results and survey results in a database and refers to this data when making the next meal suggestion.

[0898] 5. Emotion analysis

[0899] The server uses an emotion engine to analyze the user's emotions from their facial expressions and behavior, and then suggests appropriate meals based on those emotions.

[0900] 6. Collecting surveys with multiple people

[0901] When dining with multiple people, the server collects preferences and allergy information from all participants in a questionnaire format.

[0902] 7. Food suggestions

[0903] Based on the collected survey results, the server will suggest dishes suitable for all participants.

[0904] Specific examples

[0905] User A: Last time, he ordered a high-calorie pizza, and responded in the survey that "It was delicious, but I'm concerned about the calories." The emotion engine detects a "happy" facial expression.

[0906] Suggestion for next time: Suggest a low-calorie salad or grilled chicken.

[0907] Prompt Sentence Examples

[0908] User ID: 1

[0909] Last meal: Pizza

[0910] Survey results: It was delicious, but I was concerned about the calories.

[0911] Emotion data: Joy

[0912] Next meal suggestion: Low-calorie salad and grilled chicken

[0913] In this way, a system can be created that suggests next meal plans based on a user's past eating history and emotional data.

[0914] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0915] Step 1:

[0916] A user takes a photo of a meal using a smartphone or smart glasses. The input is the photo of the meal taken by the user, and the output is the photo data.

[0917] Step 2:

[0918] The device sends the photograph taken to the server. The input is the photograph data taken by the user, and the output is the photograph data sent to the server.

[0919] Step 3:

[0920] The server uses image analysis software to analyze the photos sent and identify the ingredients contained in the dish. The input is the photo data, and the output is the analyzed ingredient data. Specifically, the system uses image recognition technology and machine learning models to identify ingredients.

[0921] Step 4:

[0922] The server calculates the presence or absence of allergens and estimated calories based on the analysis results, and generates appropriate drink and food recommendations. The input is the analyzed ingredient data, and the output is allergy information, calorie information, and advice information. Specifically, it retrieves allergy information from the database and applies a calorie calculation algorithm.

[0923] Step 5:

[0924] The server saves the analysis results and survey results in a database. The input is the analysis results and survey results, and the output is the saved data. Specifically, it executes an SQL query to insert data into the database.

[0925] Step 6:

[0926] The server uses an emotion engine to analyze emotions from the user's facial expressions and behavior. The input is the user's facial expression and behavior data, and the output is the analyzed emotion data. The specific operation is to apply an emotion recognition algorithm.

[0927] Step 7:

[0928] The server suggests appropriate meals based on the emotion data. The input is the emotion data and the stored meal history data, and the output is meal suggestion data. Specifically, the server combines the emotion data and the meal history data and applies a meal suggestion algorithm.

[0929] Step 8:

[0930] When the server is hosting a meal for multiple people, it conducts a survey based on the participants' preferences. The input is the participants' preferences and allergy information, and the output is the survey result data. Specifically, it generates a survey form and sends it to the participants.

[0931] Step 9:

[0932] The server recommends dishes based on the survey results. The input is the survey result data, and the output is the recommended dish data. Specifically, the server analyzes the survey results and applies an algorithm to select the optimal dish.

[0933] Step 10:

[0934] The server sends the generated meal suggestions and advice to the user's device. The input is meal suggestion data and advice data, and the output is suggestions and advice displayed on the user's device. Specifically, the data is sent to the user's device and displayed in the application.

[0935] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0936] The data generation model 58 is a so-called generative AI (Artificial Intelligence).

[0937] An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generative AIs. Data generation models 58

[0938] , obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in the data format of voice data, text data, etc. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0939] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0940] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0941] [Second embodiment]

[0942] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0943] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0944] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0945] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0946] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0947] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0948] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0949] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0950] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0951] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0952] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0953] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0954] "Example 1"

[0955] When a user takes a photo of a meal, the system analyzes the ingredients contained in the dish from the photo. The analysis is performed using image recognition technology, specifically a deep learning-based neural network, to identify the type of dish and its ingredients from the photo. Based on the analysis results, the system sends the user advice on whether the dish contains any allergens, estimated calories, and recommended drinks and foods to accompany the dish.

[0956] "Example 2"

[0957] For multi-person dining, the system of the present invention conducts a survey based on the participants' preferences. The survey is conducted through an interface provided by the system, and participants input their preferences and dietary restrictions. Based on the survey results, the system recommends dishes. For example, if the survey reveals that all participants prefer fish dishes, the system will suggest a menu centered around fish dishes.

[0958] "Example 3"

[0959] The system of the present invention saves the analysis results and questionnaire results and makes meal suggestions based on those results for the next meal. The results are saved in a database within the user's profile and are referenced when making the next meal suggestion. For example, if a user ate a high-calorie meal in their last meal, the system can suggest a low-calorie meal for the next meal, taking into account the user's dietary balance.

[0960] The processing flow of each embodiment will be described below.

[0961] "Example 1"

[0962] Step 1: The user takes a photo of their meal.

[0963] Step 2: The system analyzes the ingredients in the dish from the photo using a deep learning-based neural network.

[0964] Step 3: Based on the analysis results, the system will send the user advice on whether they have any allergies, estimated calories, and recommended drinks and foods to ingest together.

[0965] "Example 2"

[0966] Step 1: For multi-person dining, the system surveys participants based on their preferences.

[0967] Step 2: Participants complete a questionnaire, including their preferences and dietary restrictions.

[0968] Step 3: Based on the survey results, the system will suggest recommended dishes.

[0969] "Example 3"

[0970] Step 1: The system saves the analysis results and survey results.

[0971] Step 2: The saved results will be referenced when suggesting meals from the next time onwards.

[0972] Step 3: The next time a meal recommendation is made, the system will use the saved results to make the recommendation.

[0973] Example 1

[0974] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0975] Conventional meal management systems require users to manually enter meal details, which is time-consuming and often lacks accuracy. They also struggle to provide comprehensive meal management, such as checking for allergens, calculating calories, and providing advice on accompanying foods and drinks. Furthermore, when dining with multiple people, they are unable to provide suggestions that take into account the preferences and allergy information of each participant. To address these issues, there is a need for a system that allows users to easily manage meal details and provides accurate allergy information, calorie calculations, and appropriate advice.

[0976] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0977] In this invention, the server includes a means for acquiring an image of a meal, a means for analyzing the ingredients contained in the dish from the acquired image, a means for determining the presence or absence of allergens based on the analysis results, a means for calculating estimated calories based on the analysis results, and a means for providing advice on recommended foods and beverages to ingest based on the analysis results. This allows users to easily manage their meal contents and receive accurate allergy information, calorie calculations, and appropriate advice simply by taking a photo of their meal. Furthermore, in the case of a meal with multiple people, the server further includes a means for conducting a survey based on the preferences of the participants and a means for suggesting recommended dishes based on the survey results, thereby enabling suggestions that take into account the preferences and allergy information of all participants. Furthermore, the server includes a means for saving the analysis results and survey results and making meal suggestions based on those results for future meals, facilitating ongoing dietary management.

[0978] "Means for acquiring images of meals" refers to devices or software that allow a user to take a photo of a meal and send the image data to the system.

[0979] "Means for analyzing the ingredients contained in a dish from an acquired image" refers to devices or software that use image recognition technology to identify the type of dish and its ingredients from a photograph of a meal.

[0980] "Means for determining whether or not an allergen is present" refers to devices or software that compares the analyzed material information with the allergy information registered by the user to confirm whether or not an allergen is present.

[0981] "Means for calculating estimated calories" refers to devices or software for calculating the total calories of a dish based on analyzed ingredient information.

[0982] "Means for providing advice on foods and beverages to be consumed together" refers to devices and software that suggest foods and beverages that go well with a dish based on the analysis results.

[0983] "Means for taking surveys" refers to devices or software used to collect information about preferences and allergies of participants at group meals.

[0984] "Means for suggesting recommended dishes" refers to devices or software that make recommendations based on the survey results, taking into account the preferences and allergy information of all participants.

[0985] "Means for saving analysis results and survey results" refers to devices and software for saving analyzed material information and survey results in a database or the like.

[0986] "Means for making meal suggestions for the next time and onwards" refers to devices or software for making meal suggestions for the next time and onwards based on the saved analysis results and survey results.

[0987] This invention is a system that allows a user to take a photo of a meal and analyzes the ingredients contained in the dish from the photo. The system includes means for acquiring an image of the meal, means for analyzing the ingredients contained in the dish from the acquired image, means for determining the presence or absence of allergens based on the analysis results, means for calculating estimated calories based on the analysis results, and means for providing advice on recommended foods and drinks to ingest along with the meal based on the analysis results.

[0988] Hardware and software used

[0989] Users take photos of their meals using a smartphone or camera-equipped device. The device then uploads the photos to the server. The server receives the uploaded photos and uses deep learning libraries such as TensorFlow and PyTorch to perform image analysis. The analysis results are stored in a database and notified to the user.

[0990] Specific examples

[0991] A user orders "curry rice" at a restaurant and takes a photo of it with their smartphone. The device uploads the photo to the server. The server receives the photo and performs image analysis using TensorFlow. As a result of the analysis, ingredients such as "rice, curry sauce, meat, and vegetables" are identified. The server compares this with the allergy information registered by the user to determine whether or not any allergens are present. For example, if the user has a "nut allergy," the server checks whether the ingredients contain nuts. Next, the server calculates the estimated calorie content of the dish based on the ingredients and displays "700 kcal." The server then generates advice such as "Lassi goes well with curry rice" and notifies the user.

[0992] Prompt Sentence Examples

[0993] "Analyze a user-taken photo of spaghetti bolognese and identify the ingredients. If the user has a dairy allergy, display a warning and provide food and drink recommendations along with estimated calories."

[0994] In this way, users can easily manage their meal contents, receive accurate allergy information, calorie calculations, and appropriate advice simply by taking photos of their meals. Furthermore, for meals with multiple people, including a means for conducting a survey based on the participants' preferences and a means for proposing recommended dishes based on the survey results makes it possible to make suggestions that take into account the preferences and allergy information of all participants. Furthermore, including a means for saving the analysis results and survey results and making future meal suggestions based on those results makes ongoing dietary management easier.

[0995] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0996] Step 1:

[0997] The user takes a photo of the meal.

[0998] A user takes a photo of a meal using a smartphone or other device with a camera. For example, a user orders "curry rice" at a restaurant and takes a photo of it. The input is the photo of the meal, and the output is the captured image data.

[0999] Step 2:

[1000] The device uploads the photo to the server.

[1001] The device sends the captured photos to a server via an Internet connection. The input is the captured image data, and the output is the image data uploaded to the server.

[1002] Step 3:

[1003] The server receives the photo and performs image analysis.

[1004] The server receives the photos sent from the device. It performs image analysis on the received photos using deep learning libraries such as TensorFlow and PyTorch. The input is the uploaded image data, and the output is the analyzed type of food and ingredient information. Specifically, the neural network model analyzes the photo and identifies "curry rice" and its ingredients.

[1005] Step 4:

[1006] The server identifies the material based on the analysis results.

[1007] The server identifies the type of food and its ingredients in the photo based on the results of image analysis. The input is the analyzed information on the type of food, and the output is the identified ingredient information. For example, if "curry rice" is identified, ingredients such as "rice, curry sauce, meat, and vegetables" will be listed.

[1008] Step 5:

[1009] The server checks for the presence of allergens.

[1010] The server compares the identified ingredients with the allergy information registered by the user. The input is the identified ingredient information and the user's allergy information, and the output is the determination result of whether or not the allergen is present. For example, if the user has a "nut allergy," it checks whether the ingredients contain nuts.

[1011] Step 6:

[1012] The server calculates the estimated calories.

[1013] The server calculates the estimated calories of a dish based on the identified ingredients. The input is the identified ingredient information, and the output is the calculated estimated calories. For example, in the case of "curry rice," the calories of each ingredient are added together to calculate "700 kcal."

[1014] Step 7:

[1015] The server then generates advice on recommended drinks and foods to consume.

[1016] Based on the analysis results, the server generates advice on recommended drinks and foods to ingest. The input is the analysis results and the identified ingredient information, and the output is the generated advice. For example, it generates advice such as "Lassi goes well with curry rice."

[1017] Step 8:

[1018] The server sends the analysis results and advice to the user.

[1019] The server sends the analysis results and advice to the user. The input is the analysis results and generated advice, and the output is a notification sent to the user. The user can check the results on the screen of their smartphone or device.

[1020] (Application example 1)

[1021] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1022] In today's modern diet, it is important for users to accurately understand what they are eating and manage their health. However, when eating out or using delivery services, it is difficult to understand the ingredients, allergens, and calorie information contained in the food. There is also a lack of systems that suggest appropriate drinks or additional dishes based on the user's preferences and health status. This makes it difficult for users to manage their health and increases the risk of allergic reactions and overdose.

[1023] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1024] In this invention, the server includes means for taking a photo of a meal, means for analyzing the ingredients contained in the dish from the taken photo, means for sending information on the presence or absence of allergens, estimated calories, and advice on recommended drinks and foods to accompany the dish based on the analysis results, means for taking a photo of a dish ordered by a user, analyzing the ingredients contained in the dish from the photo, and providing information on the presence or absence of allergens and calories, and means for suggesting recommended drinks and additional dishes to accompany the dish based on the user's preferences and health condition. This allows the user to accurately understand the contents of their meal and manage their health.

[1025] "Means for taking photos of meals" refers to devices or functions that allow users to take photos of meals.

[1026] "Means for analyzing ingredients contained in a dish from a photograph" refers to a device or function for identifying ingredients contained in a dish based on a photograph.

[1027] "Means for sending advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to ingest based on the analysis results" refers to devices or functions that provide information on the presence or absence of allergens and calories based on the analysis results, and recommend appropriate drinks and foods.

[1028] "Means for taking a photo of the dish ordered by the user, analyzing the ingredients contained in the dish from the photo, and providing calorie information and the presence or absence of allergens" refers to a device or function that takes a photo of the dish ordered by the user, analyzes the ingredients based on the photo, and provides calorie information and the presence or absence of allergens.

[1029] "Means for suggesting drinks and additional dishes to be consumed together based on the user's preferences and health condition" refers to a device or function for suggesting appropriate drinks and additional dishes taking into consideration the user's preferences and health condition.

[1030] The system for implementing this invention allows users to take a photo of a meal and analyze the ingredients contained in the dish from the photo. The system is realized as an application installed on a mobile device such as a smartphone or tablet.

[1031] The server uses TensorFlow and the ResNet50 model for image recognition. When a user takes a photo of a meal, the photo is sent to the server. The server preprocesses the photo and inputs it into the ResNet50 model to identify the ingredients contained in the dish. Based on the identified ingredients, the server provides information on the presence or absence of allergens and calorie content. It also suggests recommended drinks and additional dishes to accompany the meal based on the user's preferences and health status.

[1032] Specifically, the server performs the following steps: First, it receives a photo of the meal taken by the user and preprocesses the image. Next, it inputs the preprocessed image into a ResNet50 model to identify the ingredients of the dish. Based on the identified ingredients, it checks for the presence of allergens and calculates calorie information. Furthermore, it takes into account the user's preferences and health condition to suggest appropriate drinks and additional dishes.

[1033] The hardware used is a mobile device such as a smartphone or tablet, and the software uses the TensorFlow library and ResNet50 model, allowing users to accurately track their diet and manage their health.

[1034] For example, if a user takes a photo of a pizza, the server can identify the ingredients in the pizza, check for allergens (e.g., gluten), provide calorie information for the pizza, and suggest a drink (e.g., cola) to accompany it.

[1035] Example prompts to input to a generative AI model:

[1036] Analyze photos of meals taken by users to identify ingredients, identify allergens, provide calorie information, and suggest companion drinks and additional dishes.

[1037] In this way, a food delivery service application can be realized that provides information that takes into consideration the user's health and preferences.

[1038] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1039] Step 1:

[1040] The user takes a photo of the meal.

[1041] Input: A photo of a meal taken by the user on their smartphone or tablet.

[1042] Output: Photo data of the food taken.

[1043] How it works: The user takes a photo of their meal using the camera on their smartphone or tablet, which is then saved to their device.

[1044] Step 2:

[1045] The device sends the photograph to the server.

[1046] Input: Photo data of the food taken.

[1047] Output: The photo data sent to the server.

[1048] Specific operation: The device sends the captured photo data to the server via the Internet using a communication protocol such as an HTTP request.

[1049] Step 3:

[1050] The server preprocesses the received photos.

[1051] Input: Photo data sent to the server.

[1052] Output: Preprocessed image data.

[1053] Specific operation: The server performs preprocessing such as resizing and normalizing the received photo data, and converts it into a format suitable for the image recognition model.

[1054] Step 4:

[1055] The server inputs the preprocessed images into a ResNet50 model to identify the ingredients of the dish.

[1056] Input: Preprocessed image data.

[1057] Output: A list of ingredients for the identified dish.

[1058] How it works: The server inputs the preprocessed image data into the ResNet50 model, which analyzes the image and identifies the ingredients in the dish. The identified ingredients are then output in a list format.

[1059] Step 5:

[1060] The server checks for the presence of allergens based on the identified ingredients and calculates calorie information.

[1061] Input: A list of ingredients for the identified dish.

[1062] Output: Allergen information and calorie information.

[1063] Specific operation: The server refers to the specified ingredient list, checks whether it contains any allergens, and calculates the calorie count of the entire dish based on the calorie information of each ingredient.

[1064] Step 6:

[1065] The server suggests recommended drinks and additional dishes to accompany the meal based on the user's preferences and health status.

[1066] Input: Ingredient list of identified dishes, whether or not they contain allergens, calorie information, and user preferences and health data.

[1067] Output: A list of recommended drinks and additional dishes.

[1068] Specific operation: The server refers to the user's preferences and health data, and suggests suitable drinks and additional dishes based on the identified ingredients and calorie information. The suggestions are output in a list format.

[1069] Step 7:

[1070] The server sends the analysis results and suggestions to the user.

[1071] Input: allergen availability, calorie information, and a list of recommended drinks and extra dishes.

[1072] Output: Analysis results and suggestions sent to the user.

[1073] How it works: The server compiles the analysis results and recommendations and sends them to the user's device, where they can view the results on their smartphone or tablet.

[1074] Example 2

[1075] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1076] Conventional meal recommendation systems have difficulty making recommendations that fully take into account participants' preferences and dietary restrictions, making it particularly difficult to choose a menu that will satisfy everyone when dining with multiple people. Furthermore, analysis results and survey results cannot be used to make recommendations in the future, which means users have to input the same information each time.

[1077] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1078] In this invention, the server includes means for taking photos of meals, means for analyzing the ingredients contained in the dishes from the photos, means for sending information on the presence or absence of allergies, estimated calories, and advice on recommended drinks and foods to ingest based on the analysis results, means for taking a survey based on participants' preferences, means for saving the survey results, means for analyzing the saved survey results, means for suggesting recommended dishes based on prompts using a generative AI model, and means for displaying the recommendation results. This makes it possible to suggest optimal dishes taking into account the preferences and dietary restrictions of all participants, and past data can be used for future meal suggestions.

[1079] "Means for taking photos of meals" refers to devices or software that allow a user to take photos of meals and send the image data to the system.

[1080] "Means for analyzing ingredients contained in a dish from a photograph" refers to devices or software that use image analysis technology to identify ingredients contained in a dish from a photograph of the food.

[1081] "Means of sending advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to ingest based on the analysis results" refers to devices and software that calculate the presence or absence of allergens and calories based on analyzed ingredient information, and provide users with advice on appropriate drinks and foods.

[1082] "Means for surveying participants based on their preferences" refers to an interface or software used to collect dietary preferences and restrictions from multiple participants.

[1083] "Means for storing survey results" refers to devices or software for storing collected survey data in a storage device such as a database.

[1084] "Means for analyzing stored survey results" refers to any device or software used to analyze stored survey data and understand participants' preferences and limitations.

[1085] "Means for using a generative AI model to suggest recommended dishes based on a prompt sentence" refers to a device or software that inputs a prompt sentence into a generative AI model and generates optimal dish suggestions as a result.

[1086] "Means for displaying the suggestion results" refers to devices or software for visually displaying the dish suggestion results obtained from the generative AI model to the user.

[1087] The present invention is a system for proposing optimal dishes for a group meal, taking into consideration the preferences and dietary restrictions of the participants. A specific embodiment of this system will be described below.

[1088] The server first provides a questionnaire interface that users can access. This interface is built using HTML, CSS, and JavaScript and runs on a web browser. Users access this interface using their devices (PCs or smartphones) and enter their preferences and dietary restrictions.

[1089] Once the user has completed filling out the survey form, the device sends the data to the server, which processes the data using a Python script and stores it in a database (e.g., MySQL or PostgreSQL).

[1090] The server then analyzes the stored survey results using a generative AI model (e.g., OpenAI's GPT-4) and inputs the survey results as prompts into the model to generate optimal recipe suggestions.

[1091] Specifically, the following prompt sentences are input to the generative AI model:

[1092] "Based on the results of the following survey, please suggest a dish that will satisfy all participants. Survey results: Participant A likes fish dishes, Participant B is a vegetarian, and Participant C does not like spicy food."

[1093] Based on this prompt, the generative AI model will return a suggestion, such as "grilled fish and vegetable salad with a mild dressing." The server will then display this suggestion to the user, who can then view it through their browser and choose a dish based on it.

[1094] Furthermore, the server stores the analysis results and survey results and can use them to make meal suggestions for future visits, eliminating the need for users to enter the same information each time and enabling more efficient suggestions of optimal dishes.

[1095] For example, suppose participants A, B, and C at a dinner party have the following preferences and restrictions:

[1096] Participant A: I like fish dishes.

[1097] Participant B: Vegetarian

[1098] Participant C: I don't like spicy food.

[1099] In this case, the user fills in a survey form with their preferences and restrictions. The device sends the data to a server, which stores and analyzes it. The generative AI model is then given the following prompt:

[1100] "Based on the results of the following survey, please suggest a dish that will satisfy all participants. Survey results: Participant A likes fish dishes, Participant B is a vegetarian, and Participant C does not like spicy food."

[1101] The generative AI model returns suggestions such as "grilled fish with vegetable salad, served with a mild dressing," which the server displays to the user, who can then choose a dish based on the suggestions.

[1102] In this way, the invention allows for optimal dish suggestions that take into account the preferences and dietary restrictions of all participants.

[1103] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1104] Step 1: Providing a survey interface

[1105] The server provides a survey interface that users can access. This interface is built using HTML, CSS, and JavaScript and runs on a web browser. Users access this interface using their devices (PCs or smartphones).

[1106] Input: User's access request

[1107] Output: Display the survey form

[1108] Specific operation: The server provides the HTML file of the questionnaire form through the web server and displays it in the user's browser.

[1109] Step 2: Enter and submit survey data

[1110] Users fill out a questionnaire about their preferences and dietary restrictions, then click the submit button.

[1111] Input: User preferences and dietary restrictions

[1112] Output: Send survey data

[1113] Specific operation: The user enters text or options into form fields and clicks the submit button. The device uses JavaScript to collect the form data and sends it to the server as an HTTP POST request.

[1114] Step 3: Save your survey data

[1115] The server stores the received survey data in a database using Python scripts and a database management system.

[1116] Input: Survey data

[1117] Output: Save to database

[1118] What it does: The server processes the received data using a Python framework, establishes a database connection, executes SQL queries, and saves the data.

[1119] Step 4: Analyze the survey data

[1120] The server analyzes the stored survey data using a generative AI model.

[1121] Input: Saved survey data

[1122] Output: prompt statement

[1123] Specific operation: The server retrieves the survey data from the database using a Python script and generates a prompt sentence.

[1124] Step 5: Food recommendations

[1125] The generative AI model suggests dishes based on the prompt, and the server receives the suggestions.

[1126] Input: prompt statement

[1127] Output: Food recommendations

[1128] Specific operation: The server sends a prompt sentence to the generative AI model, and the generative AI model generates a suggestion result and returns it to the server.

[1129] Step 6: Viewing the Suggestion Results

[1130] The server displays the recommendations received from the generative AI model to the user, who can then choose a dish based on the recommendations.

[1131] Input: Food recommendations

[1132] Output: Display of proposal results

[1133] Specific operation: The server formats the proposal results in HTML format and provides them to the user as a web page. The user can then view the proposal results through their browser.

[1134] (Application example 2)

[1135] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1136] Conventional meal recommendation systems have difficulty considering individual user preferences and dietary restrictions, making it difficult to recommend dishes that will satisfy everyone, especially when dining with multiple people. They also lack a mechanism for effectively utilizing user preference data and reflecting it in future meal recommendations. Furthermore, there was a need for a simple interface that can be used with smartphones.

[1137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1138] In this invention, the server includes means for acquiring images of food, means for analyzing the ingredients contained in the food from the acquired images, means for providing, based on the analysis results, the presence or absence of allergens, estimated calories, and advice on recommended beverages and foods to ingest together, means for conducting a survey based on the preferences of participants in the case of a meal with multiple people, means for suggesting recommended dishes based on the survey results, means for saving the survey results and using those results to suggest meals for the next time and thereafter, means for collecting user preferences and dietary restrictions through an application installed on a smartphone and suggesting optimal dishes based on shared preferences, and means for analyzing user preference data using a generative AI model and generating prompt sentences, thereby making it possible to suggest dishes that will satisfy everyone in a meal with multiple people.

[1139] The "means for acquiring an image of a meal" is a function that allows a user to take a photo of a meal and send the image data to the system.

[1140] The "means for analyzing the ingredients contained in a dish from an acquired image" is a function that uses image analysis technology to identify the ingredients contained in a dish from a photograph of the food.

[1141] "Means for providing advice on the presence or absence of allergens, estimated calories, and recommended beverages and foods to ingest together" is a function that calculates the presence or absence of allergens and estimated calories based on analyzed ingredient information, and also provides users with advice on appropriate beverages and foods.

[1142] "In the case of a meal with multiple people, a means of conducting a survey based on the preferences of the participants" is a function that collects information about the preferences and dietary restrictions of multiple users in the form of a survey.

[1143] "Means for suggesting recommended dishes based on survey results" is a function that analyzes collected survey results and suggests dishes that will satisfy everyone.

[1144] "Means for saving survey results and making meal suggestions based on those results" is a function that saves the collected survey results in a database and uses them to make meal suggestions for the next time and onwards.

[1145] "A means of collecting user preferences and dietary restrictions through an application installed on a smartphone and suggesting optimal dishes based on shared preferences" refers to a function that uses a smartphone application to allow users to input their preferences and dietary restrictions, finds shared preferences based on that data, and suggests optimal dishes.

[1146] "Means for analyzing user preference data using a generative AI model and generating prompt sentences" refers to a function that uses a generative AI model to analyze user preference data and generate appropriate prompt sentences based on the results.

[1147] As an embodiment of the present invention, the following system is constructed.

[1148] System Configuration

[1149] Hardware:

[1150] Smartphone

[1151] server

[1152] Database (SQLite)

[1153] software:

[1154] Smartphone application (iOS / Android)

[1155] Server-side programs (Python, Flask)

[1156] Image analysis software (OpenCV)

[1157] Generative AI models (e.g., GPT-3)

[1158] Processing flow

[1159] 1. How to get food images:

[1160] Users take photos of their meals using a smartphone app, and the images are sent to a server via the app.

[1161] 2. A method for analyzing the ingredients contained in a dish from the captured image:

[1162] The server analyzes the received images using image analysis software (OpenCV) and identifies the ingredients contained in the dish.

[1163] 3. Means of providing advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to ingest together:

[1164] Based on the analysis results, the system calculates the presence or absence of allergens and estimated calories, and provides users with advice on appropriate beverages and foods.

[1165] 4. For multi-person dining, survey participants based on their preferences:

[1166] For meals with multiple people, a survey about each participant's preferences and dietary restrictions is conducted via a smartphone app.

[1167] 5. How to suggest dishes based on survey results:

[1168] The server analyzes the collected survey results and suggests dishes that will satisfy everyone.

[1169] 6. A means to store survey results and use them to make future meal recommendations:

[1170] The survey results are stored in a database (SQLite) and will be used to make meal suggestions from next time onwards.

[1171] 7. A means of collecting user preferences and dietary restrictions through an application installed on a smartphone and suggesting the best dishes based on shared preferences:

[1172] The smartphone app collects users' preferences and dietary restrictions and suggests the best dishes based on shared tastes.

[1173] 8. Using a generative AI model to analyze user preference data and generate prompts:

[1174] The server uses a generative AI model (such as GPT-3) to analyze user preference data and generate appropriate prompts based on the results.

[1175] Specific examples

[1176] For example, if participant A likes "sushi," participant B likes "sushi," and participant C likes "sushi," the system will suggest "sushi." Allergen and calorie information will also be provided.

[1177] Example prompt sentence:

[1178] "You can enter your preferences and dietary restrictions. For example, your favorite food is 'sushi,' your least favorite food is 'green peppers,' and you have no dietary restrictions. The system will then use this information to suggest dishes that will please everyone."

[1179] In this way, users can easily have meals that suit their preferences suggested.

[1180] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1181] Step 1:

[1182] Users take photos of their meals using a smartphone app.

[1183] Input: Food photo

[1184] Output: Captured image data

[1185] Specific operation: The user takes a photo of a meal using the camera function of their smartphone and saves the image data in the application.

[1186] Step 2:

[1187] The terminal transmits the captured image data to the server.

[1188] Input: Captured image data

[1189] Output: Image data sent to the server

[1190] Specific operation: The smartphone app calls an API to upload image data to the server.

[1191] Step 3:

[1192] The server analyzes the received images using image analysis software (OpenCV) and identifies the ingredients contained in the dish.

[1193] Input: Image data sent to the server

[1194] Output: Parsed ingredient information

[1195] Specific operation: The server uses OpenCV to analyze the image and executes an algorithm to identify the ingredients contained in the dish.

[1196] Step 4:

[1197] Based on the analysis results, the server calculates the presence or absence of allergens and estimated calories, and provides the user with advice on appropriate drinks and foods.

[1198] Input: Parsed ingredient information

[1199] Output: Allergens, estimated calories, drink and food recommendations

[1200] Specific operation: The server checks for the presence of allergens based on the ingredient information, calculates calories, and generates advice on appropriate drinks and foods.

[1201] Step 5:

[1202] If dining with multiple people, the device will conduct a survey about each participant's preferences and dietary restrictions.

[1203] Input: Information about each participant's preferences and dietary restrictions

[1204] Output: Survey results

[1205] How it works: The smartphone app displays a survey form where each participant enters their preferences and dietary restrictions.

[1206] Step 6:

[1207] The server analyzes the collected survey results and suggests dishes that will satisfy everyone.

[1208] Input: Survey results

[1209] Output: Recommended dishes

[1210] What it does: The server analyzes the survey results, identifies common preferences, and runs an algorithm to suggest the best dishes based on those preferences.

[1211] Step 7:

[1212] The server stores the survey results in a database (SQLite) and uses them to make meal suggestions for future visits.

[1213] Input: Survey results

[1214] Output: Survey results stored in a database

[1215] Specific operation: The server saves the survey results in an SQLite database and performs database operations to use them for future proposals.

[1216] Step 8:

[1217] The server uses a generative AI model (such as GPT-3) to analyze user preference data and generate appropriate prompts based on the results.

[1218] Input: User preference data

[1219] Output: prompt statement

[1220] Specific operation: The server calls the generative AI model, analyzes the user's preference data as input, and generates an appropriate prompt sentence.

[1221] Example prompt sentence:

[1222] "You can enter your preferences and dietary restrictions. For example, your favorite food is 'sushi,' your least favorite food is 'green peppers,' and you have no dietary restrictions. The system will then use this information to suggest dishes that will please everyone."

[1223] Example 3

[1224] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1225] Conventional meal recommendation systems have the problem that it is difficult to make optimal meal recommendations for each individual user because they do not fully utilize the user's past eating history or questionnaire results. In addition, they do not take into consideration the balance of meals, and therefore cannot contribute to the user's health management. Furthermore, when dining with multiple people, there is the problem that they cannot make recommendations based on the preferences of all participants.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1227] In this invention, the server includes a means for inputting the results of a questionnaire about the user's meals, a means for transmitting the inputted results of the questionnaire to the server, a means for storing the received results of the questionnaire in a database, a means for generating meal suggestions for the next meal or later based on the stored data, and a means for displaying the generated meal suggestions to the user. This enables personalized meal suggestions based on the user's past meal history and the questionnaire results, contributing to the user's health management. Furthermore, even in a multi-person meal, suggestions based on the preferences of all participants can be made.

[1228] "Means for taking photos of meals" refers to devices or functions that allow users to take photos of meals.

[1229] "Means for analyzing ingredients contained in a dish from a photograph" refers to a device or function for analyzing a photograph and identifying the ingredients contained in a dish.

[1230] "Means of sending advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to ingest based on the analysis results" refers to devices and functions that determine the presence or absence of allergens and estimated calories based on the analysis results, and recommend appropriate drinks and foods.

[1231] The term "means for inputting the results of a questionnaire about the user's diet" refers to a device or function that allows the user to answer a questionnaire about diet and input the results.

[1232] "Means for transmitting the entered survey results to the server" refers to a device or function for transmitting the survey results entered by the user to the server.

[1233] "Means for storing the survey results received by the server in a database" refers to a device or function for storing the survey results received by the server in a database.

[1234] "Means for generating meal suggestions for the next time and thereafter based on the stored data" refers to a device or function for generating meal suggestions for the next time and thereafter based on the data stored in the database.

[1235] "Means for displaying the generated meal suggestions to the user" refers to a device or function for displaying the generated meal suggestions to the user.

[1236] "Means for conducting a survey based on the preferences of participants when dining with multiple people" refers to a device or function for conducting a survey based on the preferences of participants when dining with multiple people.

[1237] "Means for suggesting recommended dishes based on survey results" refers to devices or functions for suggesting recommended dishes based on survey results.

[1238] The present invention relates to a system for suggesting meals for the next time and thereafter based on the results of a questionnaire and analysis of meals of a user. A specific embodiment of this system will be described below.

[1239] Hardware and software used

[1240] Hardware

[1241] Device: A device used by users to input meal information, such as a smartphone, tablet, or computer.

[1242] Server: A computer system that receives, stores, and analyzes data sent by users.

[1243] Database: A storage system connected to the server for storing users' dietary information and survey results.

[1244] software

[1245] Survey entry application: An application that allows users to fill out surveys about their diet.

[1246] Data transmission module: A software module for transmitting data from a terminal to a server.

[1247] Database Management System: Software that allows a server to store and manage data.

[1248] Generative AI model: An artificial intelligence model that generates next meal suggestions based on the user's past eating history and survey results.

[1249] System Operation

[1250] User Input

[1251] The user answers a questionnaire about their diet using a questionnaire entry application on the terminal. For example, the user enters, "I had a hamburger and fries for lunch today." This information is recorded as the user's diet history.

[1252] Sending data

[1253] The terminal transmits the meal information entered by the user to the server via the data transmission module. At this time, the terminal converts the data into an appropriate format and transmits it to the server. For example, data such as "Lunch: Hamburger and fries" is transmitted to the server.

[1254] Data storage

[1255] The server uses a database management system to store the received meal information in a database. When storing the information, metadata such as the user ID and date and time are also stored. For example, the information is stored in the database in the format "User ID: 12345, Lunch: Hamburger and fries, Date and time: 2023-10-01."

[1256] Next meal suggestion

[1257] The next time the server makes a meal suggestion, it references the database and generates a new meal suggestion based on the user's past eating history. For example, if the user previously ate a high-calorie meal, the server uses the generative AI model to suggest a low-calorie meal.

[1258] Submit a proposal

[1259] The server sends the generated meal suggestions to the device, for example, a message saying "I recommend salad and soup for your next meal."

[1260] View Suggestions

[1261] The device displays the meal suggestions received from the server to the user. For example, a message such as "We recommend salad and soup for your next meal" may be displayed on the device screen.

[1262] Examples and prompts

[1263] Specific examples

[1264] 1. The user types into the terminal, "I had a hamburger and fries for lunch today."

[1265] 2. The device sends this information to the server.

[1266] 3. The server stores this information in a database.

[1267] 4. When suggesting your next meal, the server will refer to the database and suggest, "We recommend salad and soup for your next meal."

[1268] 5. The device displays this suggestion to the user.

[1269] Prompt Sentence Examples

[1270] "Please explain the process of the system that suggests a low-calorie meal for the next meal if the user had a high-calorie meal in the previous meal." The flow of the specific process in Example 3 will be explained using Figure 15.

[1271] Step 1:

[1272] The user answers a questionnaire about meals using a questionnaire input application on the terminal.

[1273] Input: The user types, "I had a hamburger and fries for lunch today."

[1274] Output: The survey results are saved to your device.

[1275] Specific actions: The user opens the application, enters meal details into the text box, and presses the submit button.

[1276] Step 2:

[1277] The terminal transmits the meal information input by the user to the server via the data transmission module.

[1278] Input: Survey results (e.g., "Lunch: Burger and fries").

[1279] Output: Meal information is sent to the server.

[1280] Specific operation: The terminal converts the survey results into JSON format and sends them to the server using an HTTP request.

[1281] Step 3:

[1282] The server stores the received meal information in a database using a database management system.

[1283] Input: Meal information received by the server (e.g., "Lunch: hamburger and fries").

[1284] Output: Meal information stored in a database.

[1285] Specific operation: The server analyzes the received data, adds metadata such as the user ID and date and time, and saves it in the database.

[1286] Step 4:

[1287] The next time the server makes a meal suggestion, it references the database and generates a new meal suggestion based on the user's past meal history.

[1288] Input: Past dietary history stored in a database.

[1289] Output: Generated meal suggestions (e.g., "I recommend salad and soup for your next meal").

[1290] Specific operation: The server uses the generative AI model to analyze past dietary data and generate suggestions that take into account the user's health condition and dietary balance.

[1291] Step 5:

[1292] The server transmits the generated meal suggestions to the terminal.

[1293] Input: Generated meal suggestions (e.g., "I recommend salad and soup for your next meal").

[1294] Output: Meal suggestions are sent to the device.

[1295] Specific operation: The server converts the generated proposal into JSON format and sends it to the device using an HTTP response.

[1296] Step 6:

[1297] The terminal displays the meal suggestions received from the server to the user.

[1298] Input: Meal suggestions received from the server (e.g., "I recommend salad and soup for your next meal").

[1299] Output: Meal suggestions displayed to the user.

[1300] Specific operation: The device analyzes the received data and displays it on the user interface, allowing the user to check the suggestions on the screen.

[1301] (Application example 3)

[1302] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1303] Conventional meal recommendation systems have the problem that they are unable to fully utilize the user's dietary history and survey results, making it difficult to make individually optimized meal recommendations. Furthermore, since the system does not take into account the user's dietary balance, there is also the problem of inadequate health management. Furthermore, there are cases where the suggested meal menu is not displayed properly to the user, resulting in a poor user experience.

[1304] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1305] In this invention, the server includes: a means for taking a photo of a user's meal; a means for analyzing the ingredients contained in the dish from the photograph; a prompt for generating advice regarding the presence or absence of allergens, estimated calories, and recommended accompanying drinks or foods based on the analysis results; and a means for generating and presenting the advice using a generative AI model. The server also includes a means for storing the user's diet history and questionnaire results in a database; a prompt for analyzing the stored diet history or questionnaire results, evaluating the user's dietary balance, and suggesting a next meal; and a means for presenting the next meal suggestion using the generative AI model. This enables individually optimized meal suggestions based on the user's diet history and questionnaire results, improving the user's health management and improving the user experience by displaying appropriate meal menus.

[1306] "Means for taking photos of meals" refers to devices or functions that allow users to take photos of meals.

[1307] "Means for analyzing ingredients contained in a dish from a photograph" refers to a device or function for analyzing a photograph of a meal and identifying the ingredients and components contained therein.

[1308] "Means for sending advice on the presence or absence of allergens, estimated calories, and recommended foods and drinks to ingest based on the analysis results" refers to devices and functions that estimate the presence or absence of allergens and calories based on analyzed data, and provide advice on appropriate foods and drinks.

[1309] "Means for saving a user's dietary history and survey results in a database" refers to a device or function for saving information about the meals a user has eaten in the past and survey results in a database.

[1310] The "means for analyzing stored data and evaluating the user's dietary balance" refers to a device or function for analyzing stored dietary history and questionnaire results and evaluating the user's dietary balance.

[1311] The "means for suggesting the next meal based on the analysis results" refers to a device or function for suggesting the next meal based on the analyzed data.

[1312] The "means for displaying a suggested meal menu to a user" refers to a device or function for visually displaying a suggested meal menu to a user.

[1313] A system for implementing the present invention has the following configuration.

[1314] System Configuration

[1315] 1. A way to take photos of your food

[1316] Users take photos of their meals using their smartphones or smart glasses, which allows the contents of the meal to be captured as digital data.

[1317] 2. A method for analyzing ingredients contained in a dish from a photograph

[1318] The server receives the photographs and uses image analysis software (e.g., OpenCV) to identify the ingredients contained in the dish, allowing for the extraction of meal composition information.

[1319] 3. A method to send advice on whether or not you have any allergies, estimated calories, and recommended drinks and foods to ingest based on the analysis results

[1320] The server uses the analysis results to estimate the presence or absence of allergens and calories, and generates appropriate food and drink recommendations, referencing nutritional information stored in a database.

[1321] 4. A means to store user's meal history and survey results in a database

[1322] The server stores the user's meal history and survey results in a database (e.g., SQLite), allowing past data to be accumulated and used for future recommendations.

[1323] 5. A means of analyzing stored data and assessing the user's dietary balance

[1324] The server analyzes the stored data and evaluates the user's dietary balance using a data analysis library (e.g., Pandas).

[1325] 6. A method to suggest your next meal based on the analysis results

[1326] The server then uses the analysis results to suggest the next meal, and uses a generative AI model (e.g., TensorFlow) to generate the optimal meal menu for the user.

[1327] 7. How to display meal suggestions to users

[1328] The proposed meal menu is displayed on the user's device (smartphone or smart glasses) using a mobile app development framework (e.g., Flutter (registered trademark)).

[1329] Specific examples

[1330] 1. Data Collection

[1331] Users take a photo of their meal with their smartphone and upload it to the app, where they are then asked to fill out a questionnaire to assess their satisfaction with the meal and their physical condition.

[1332] 2. Data Analysis

[1333] The server analyzes the ingredients contained in the dish from the uploaded photo, and generates and presents advice using a generative AI model and prompts that instruct the server to generate advice regarding the presence or absence of allergens, estimated calories, and recommended drinks or foods to ingest based on the analysis results.

[1334] Example prompt sentence: After receiving the analysis results of the ingredients contained in the dish as input, "generate advice on the presence or absence of allergens, estimated calories, and recommended drinks or foods to ingest based on the analysis results of the ingredients."

[1335] "Since your last meal was high in calories, I would like to suggest a low-calorie meal next time."

[1336] 3. Meal suggestions

[1337] The server stores the user's diet history or questionnaire results in a database. The server analyzes the stored data, evaluates the user's dietary balance, and uses a generative AI model and a prompt to suggest the next meal.

[1338] Example prompt sentence: "User's food history data: 2023-10-01: Hamburgers, fries, high calories 2023-10-02: Salad, low calorie User survey results: 2023-10-01: Satisfaction: High, Health: Normal 2023-10-02: Satisfaction: Average, Health: Good Evaluate the user's dietary balance and suggest low-calorie menus for their next meal.

[1339] Sample meal suggestions for your next meal: "I suggest a salad and grilled chicken for your next meal."

[1340] 4. User Interface

[1341] The suggested menu is displayed on a smartphone or smart glasses.

[1342] In this way, a system can be built that makes suggestions for the next meal based on the user's eating history and survey results.

[1343] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1344] Step 1:

[1345] The user takes a photo of the meal using a smartphone or smart glasses.

[1346] Input: Food photo

[1347] Output: Captured photo data

[1348] Specific operation: The user takes a photo of the meal and uploads it to the server through the application.

[1349] Step 2:

[1350] The server receives the photograph and uses image analysis software (e.g., OpenCV) to identify the ingredients contained in the dish.

[1351] Input: Photographed photo data

[1352] Output: Analyzed material data

[1353] What it does: The server analyzes the photo data and applies image recognition algorithms to identify ingredients in the image.

[1354] Step 3:

[1355] Based on the analysis results, the server estimates the presence or absence of allergens and calories, and generates appropriate food and drink advice.

[1356] Input: Analyzed material data

[1357] Output: Presence or absence of allergens, estimated calories, food and drink advice

[1358] Specific operation: The server references nutritional information stored in the database and compares it with the analysis results to estimate the presence or absence of allergens and calories, and also generates appropriate food and drink recommendations.

[1359] Step 4:

[1360] The server stores the user's meal history and survey results in a database (e.g., SQLite).

[1361] Input: User's dietary history, survey results

[1362] Output: Saved data

[1363] Specific operation: The server receives the meal history and survey results entered by the user and stores them in a database.

[1364] Step 5:

[1365] The server analyzes the stored data and evaluates the user's dietary balance.

[1366] Input: Saved data

[1367] Output: Dietary balance evaluation results

[1368] Specific operation: The server uses a data analysis library (e.g., Pandas) to analyze the stored data and evaluate the user's dietary balance.

[1369] Step 6:

[1370] The server will make suggestions for the next meal based on the analysis results.

[1371] Input: Dietary balance assessment results

[1372] Output: Next meal suggestion

[1373] Specific operation: The server uses a generative AI model (e.g., TensorFlow) to generate the optimal meal menu for the user.

[1374] Step 7:

[1375] The suggested meal menu is displayed on the user's device (smartphone or smart glasses).

[1376] Input: Next meal suggestion

[1377] Output: Displayed meal menu

[1378] Specific operation: The server sends the proposed menu to the user's terminal and displays it visually through the application.

[1379] In addition, the emotion engine described below may be used to recognize the user's emotions, and advice may be generated and presented using a prompt sentence that instructs the system to generate advice based on the analysis results and the user's emotions, and a generative AI model.

[1380] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1381] "Example 1"

[1382] One embodiment of the present invention is a system that combines an emotion engine that recognizes the user's emotions. When a user takes a photo of a meal, this system analyzes the ingredients contained in the dish from the photo. Based on the analysis results, the system sends the user advice on whether or not there are any allergies, estimated calories, and recommended drinks and foods to accompany the meal. Furthermore, the emotion engine reads emotions from the user's facial expressions and tone of voice and makes meal suggestions that match those emotions. For example, if a user shows a happy expression, the emotion engine recognizes that joy and suggests meals that the user is likely to like.

[1383] "Example 2"

[1384] Another embodiment is a system that utilizes an emotion engine even when dining with multiple people. This system conducts a survey based on the participants' preferences and suggests dishes based on the survey results. Furthermore, the emotion engine reads the emotions of all participants and suggests meals that match those emotions. For example, if all participants look happy, the emotion engine will recognize that happiness and suggest meals that fit the happy atmosphere.

[1385] "Example 3"

[1386] Another example is a system that stores analysis results and survey results and uses them to suggest future meals. This system uses an emotion engine to read the user's emotions and suggests meals that match those emotions. For example, if the user showed a happy expression the last time they ate a particular dish, that information is saved and used as a reference when suggesting the next meal.

[1387] The processing flow of each embodiment will be described below.

[1388] "Example 1"

[1389] Step 1: The user takes a photo of their meal.

[1390] Step 2: The system analyzes the ingredients in the dish from the photo.

[1391] Step 3: Based on the analysis results, the user is informed of any allergies, estimated calories, and recommended drinks and foods to ingest.

[1392] Step 4: The emotion engine reads the user's emotions from their facial expressions and tone of voice.

[1393] Step 5: The emotion engine makes meal suggestions that match the emotions it has read.

[1394] "Example 2"

[1395] Step 1: The system surveys participants based on their preferences.

[1396] Step 2: Suggest recommended dishes based on the survey results.

[1397] Step 3: The emotion engine reads the emotions of all participants.

[1398] Step 4: The emotion engine makes meal suggestions that match the emotions it has read.

[1399] "Example 3"

[1400] Step 1: The system saves the analysis results and survey results.

[1401] Step 2: Based on the saved results, suggestions for future meals are made.

[1402] Step 3: The emotion engine reads the user's emotions.

[1403] Step 4: The emotion engine makes meal suggestions that match the emotions it has read.

[1404] Example 1

[1405] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1406] With conventional dietary analysis systems, even if a user took a photo of their meal, it was difficult to accurately analyze the ingredients contained in the dish from that photo. Furthermore, it was not possible to provide information on the presence or absence of allergens and estimated calories based on the analysis results, or to make meal suggestions based on the user's emotions. Furthermore, when dining with multiple people, it was difficult to make suggestions based on the preferences of the participants.

[1407] The identification process by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes means for taking a photo of the meal, means for transmitting the taken photo to the server, means for analyzing the transmitted photo, means for generating, based on the analysis results, information on the presence or absence of allergens, estimated calories, and advice on recommended drinks and foods to ingest, means for transmitting the generated advice to the user, means for recognizing the user's emotions, and means for suggesting meals based on the recognized emotions. This makes it possible to accurately analyze ingredients from a photo of the meal taken by the user and provide information on the presence or absence of allergens and estimated calories, as well as to suggest meals based on the user's emotions. Furthermore, when dining with multiple people, suggestions can be made based on the preferences of the participants.

[1408] The "means for taking photos of meals" refers to a device with a camera function that allows the user to take photos of meals.

[1409] The "means for transmitting the photographed photo to the server" is a device equipped with a communication function for transmitting the photographed photo data to the server via the Internet.

[1410] The "means for analyzing the transmitted photographs" refers to software that uses image recognition technology to analyze the photographic data transmitted to the server and identify the type of food and its ingredients.

[1411] "Means for generating advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to ingest based on the analysis results" refers to software equipped with an algorithm for generating advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods based on the analyzed data.

[1412] The "means for transmitting the generated advice to the user" is a device having a communication function for transmitting the generated advice to the user's terminal.

[1413] The "means for recognizing user emotions" is software that uses emotion analysis technology to analyze the user's facial expressions and tone of voice and recognize their emotions.

[1414] The "means for making meal suggestions based on recognized emotions" is software having an algorithm for making appropriate meal suggestions based on the recognized emotions of the user.

[1415] The "means for conducting a survey based on the preferences of participants when dining with multiple people" is a device equipped with a survey function for collecting the preferences of participants when dining with multiple people.

[1416] The "means for suggesting recommended dishes based on survey results" is software equipped with an algorithm for suggesting dishes suitable for participants based on the collected survey results.

[1417] "Means for saving analysis results and survey results and making meal suggestions based on those results" refers to software with data storage and suggestion functions that saves analysis results and survey results in a database and can be used to make meal suggestions for the next time and onwards.

[1418] This invention is a system that allows users to take photos of their meals and analyzes the ingredients contained in the dishes from the photos. By combining image recognition technology and emotion analysis technology, this system can provide more personalized meal recommendations to users.

[1419] Hardware and software used

[1420] Hardware: Devices such as smartphones and tablets, and servers

[1421] Software: Image recognition technology (e.g., TensorFlow, PyTorch), emotion engine (e.g., Affectiva)

[1422] Specific operation of the system

[1423] 1. User takes a photo of their meal:

[1424] The user opens the camera app on their smartphone or tablet and takes a photo of a meal. For example, the user takes a photo of curry rice.

[1425] 2. The device sends the photo to the server:

[1426] The device sends the captured photo to the server via the Internet, specifically by using an HTTP POST request to upload the photo data to the server.

[1427] 3. The server analyzes the image:

[1428] The server then analyzes the received photos using image recognition technology. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to input the photos into a neural network model to identify the type of dish and its ingredients. For example, a photo of curry rice could be analyzed to identify ingredients such as chicken, potatoes, carrots, and onions.

[1429] 4. The server generates the analysis results:

[1430] Based on the analysis results, the server generates advice on whether or not there are any allergies, estimated calories, and recommended drinks and foods to ingest together. For example, it may determine that curry rice does not contain any allergens, has an estimated calorie count of 500 kcal, and that the recommended drink is green tea.

[1431] 5. The server sends the analysis results to the user:

[1432] The server sends the generated analysis results to the user's device. Specifically, it returns the analysis results as an HTTP response.

[1433] 6. Emotion engine recognizes user emotions:

[1434] The emotion engine analyzes facial expressions and tone of voice when a user takes a photo to recognize the user's emotions. For example, it uses Affectiva's SDK to determine whether the user is showing a happy expression.

[1435] 7. Servers make emotion-based meal suggestions:

[1436] The server then suggests meals that match the user's emotions based on the analysis results of the emotion engine. For example, if the user shows a happy expression, the server will suggest a dessert that the user would like (e.g., chocolate cake).

[1437] Specific examples

[1438] User photo: Curry rice photo

[1439] Analysis result: Curry rice, ingredients (chicken, potato, carrot, onion)

[1440] Advice: No allergens, estimated calorie intake of 500kcal, recommended drink: green tea

[1441] Emotion recognition: If the user shows a happy expression, suggest desserts they might like (e.g., chocolate cake)

[1442] Prompt Sentence Examples

[1443] "I took a photo of curry rice. I'd like you to analyze the ingredients, tell me if there are any allergies, the estimated calories, and what drinks and foods I should eat with it. I'd also like you to make meal suggestions based on the user's emotions."

[1444] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1445] Step 1:

[1446] The user takes a photo of their meal.

[1447] Input: A user opens the camera app on their smartphone or tablet and takes a photo of their food.

[1448] Specific operation: The user opens the camera app, takes a photo of curry rice, and then taps the "Start analysis" button in the app.

[1449] Output: Captured photo data (JPEG format).

[1450] Step 2:

[1451] The device sends the photo to the server.

[1452] Input: Captured photo data (JPEG format).

[1453] Specific operation: The device sends the captured photo to the server via the Internet. Specifically, it uploads the photo data to the server using an HTTP POST request. A progress bar is displayed during the transmission, and a "Transmission complete" message is displayed once the transmission is complete.

[1454] Output: The photo data sent to the server.

[1455] Step 3:

[1456] The server analyzes the image.

[1457] Input: Photo data sent to the server.

[1458] How it works: The server temporarily stores the received photo and then uses a deep learning framework such as TensorFlow or PyTorch to input the photo into a neural network model. The model analyzes the photo and identifies the type of dish and its ingredients. For example, a photo of curry rice could be analyzed to identify ingredients such as chicken, potatoes, carrots, and onions. The analysis takes a few seconds.

[1459] Output: Analysis results (type of food and ingredients).

[1460] Step 4:

[1461] The server generates the analysis results.

[1462] Input: Analysis results (type of food and ingredients).

[1463] Specific operation: Based on the analysis results, the server generates advice on whether or not there are any allergies, the estimated calories, and the recommended drinks and foods to ingest together. For example, it generates the information that curry rice does not contain any allergens, has an estimated calorie count of 500 kcal, and the recommended drink is green tea.

[1464] Output: Generated advice (presence of allergies, estimated calories, recommended drinks and foods).

[1465] Step 5:

[1466] The server sends the analysis results to the user.

[1467] Input: Generated advice (presence of allergies, estimated calories, recommended drinks and foods).

[1468] Specific operation: The server sends the generated analysis results to the user's device. Specifically, it returns the analysis results as an HTTP response.

[1469] Output: Analysis results sent to the user's device.

[1470] Step 6:

[1471] The emotion engine recognizes the user's emotions.

[1472] Input: The user's facial expression and tone of voice.

[1473] How it works: The emotion engine analyzes facial expressions and tone of voice when a user takes a photo to recognize the user's emotions. For example, it uses Affectiva's SDK to determine whether the user is showing a happy expression.

[1474] Output: Recognized emotion data (e.g., happiness, sadness, surprise, etc.).

[1475] Step 7:

[1476] Servers make emotion-based meal suggestions.

[1477] Input: Recognized emotion data.

[1478] Specific operation: The server makes meal suggestions that match the user's emotions based on the analysis results of the emotion engine. For example, if the user shows a happy expression, the server will suggest a dessert that the user would like (e.g., chocolate cake).

[1479] Output: Emotion-based meal suggestions.

[1480] (Application example 1)

[1481] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1482] Conventional dietary analysis systems allow users to take photos of their meals and analyze the ingredients contained in the dish, but they lack functionality such as suggesting meals based on the user's emotions, detecting allergies, and calculating estimated calories. Another problem is that they are unable to make suggestions based on the preferences of participants when dining with others. Furthermore, they lack the ability to save analysis and survey results and use them for future meal suggestions. To solve these problems, a system with more advanced analysis capabilities and a suggestion function based on the user's emotions is needed.

[1483] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1484] In this invention, the server includes means for taking photos of meals, means for analyzing the ingredients contained in the dishes from the photos, means for sending information on the presence or absence of allergens, estimated calories, and recommendations on recommended drinks and other foods to accompany the dishes based on the analysis results, means for recognizing the user's emotions, means for suggesting additional dishes and drinks based on the recognized emotions, means for conducting a survey based on the preferences of participants in the case of a meal with multiple people, means for suggesting recommended dishes based on the survey results, and means for saving the analysis results and the survey results and making meal suggestions based on these results for future meals. This enables meal suggestions based on the user's emotions, detection of allergens, and calculation of estimated calories, making it possible to make suggestions based on the preferences of participants even in a meal with multiple people. The analysis results and the survey results can also be saved and used for future meal suggestions.

[1485] "Means for taking photos of meals" refers to devices or functions that allow users to take photos of meals, and includes photography devices such as smartphones and cameras.

[1486] "Means for analyzing the ingredients contained in a dish from a photograph" refers to image recognition technology and algorithms for identifying the ingredients contained in a dish based on a photograph.

[1487] "A means of sending advice on whether or not an item contains allergens, estimated calories, and recommended drinks and foods to ingest based on the analysis results" is a function that calculates whether or not an item contains allergens and calories based on analyzed ingredient information, and provides users with advice on appropriate foods and drinks.

[1488] "Means of recognizing user emotions" refers to technologies and algorithms for reading emotions from a user's facial expressions, tone of voice, etc.

[1489] "Means for suggesting additional dishes and drinks based on recognized emotions" is a function for suggesting appropriate dishes and drinks according to the user's emotions.

[1490] "In the case of a meal with multiple people, a means of conducting a survey based on the preferences of the participants" refers to a survey function for collecting the preferences of each participant in a meal with multiple people.

[1491] "Means for suggesting recommended dishes based on survey results" is a function for suggesting dishes suitable for all participants based on the collected survey results.

[1492] "Means for saving analysis results and survey results and making meal suggestions for the next time and beyond based on those results" refers to a function that saves past analysis results and survey results in a database and makes meal suggestions for the next time and beyond based on that data.

[1493] A system for implementing this invention has the ability to allow a user to take a photo of a meal and analyze the ingredients contained in the dish from that photo. The analysis is performed using image recognition technology, specifically a deep learning-based neural network. Based on the analysis results, the system sends the user advice on whether or not the dish contains any allergens, estimated calories, and recommended drinks and foods to accompany the dish. The system also has the ability to recognize the user's emotions and suggest additional dishes and drinks based on those emotions.

[1494] Hardware and software used

[1495] Hardware: Smartphones, cameras

[1496] Software: Python, Keras, OpenCV, EmotionRecognizer library

[1497] Data processing and calculation

[1498] Image Recognition

[1499] Users take a photo of their meal with their smartphone camera and upload it to the app, where the server analyzes the photo using Keras image recognition models to identify the ingredients in the dish.

[1500] Allergen detection

[1501] The server detects whether or not an allergen is present based on the analysis results, and notifies the user if an allergy is present.

[1502] Calorie Counting

[1503] The server calculates the estimated calories from the ingredients of the dish and displays them to the user.

[1504] emotion recognition

[1505] The EmotionRecognizer library is used to read the user's emotions from their facial expressions and tone of voice, and the server then suggests appropriate dishes and drinks based on the user's emotions.

[1506] Dining with multiple people

[1507] When dining with multiple people, the server will conduct a survey based on the participants' preferences and then recommend dishes based on the results.

[1508] Data storage and next proposal

[1509] The server stores the analysis results and survey results in a database and uses this data to make meal suggestions for the next time and beyond.

[1510] Specific examples

[1511] When a user uploads a photo of a pizza they have taken with their smartphone to the app, the server analyzes the ingredients (cheese, tomato, pepperoni, etc.) contained in the pizza and displays the presence of allergens and estimated calories. Furthermore, if the user shows a happy expression, the server will suggest, "Would you like dessert?"

[1512] Prompt Sentence Examples

[1513] Analyze photos of food taken by users to identify the ingredients contained in them, calculate allergens and estimated calories and notify the user, and recognize the user's emotions and suggest additional dishes or drinks that match those emotions.

[1514] The above is an embodiment of the present invention.

[1515] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1516] Step 1:

[1517] Users take photos of their meals using their smartphone camera and upload them to the app.

[1518] Input: Food photo

[1519] Output: Uploaded photo data

[1520] Specific operation: A user takes a photo of a meal using the smartphone camera and uploads the photo to the application, which then sends the photo data to the server.

[1521] Step 2:

[1522] The server receives the uploaded photo and uses image recognition models to analyze the ingredients contained in the dish.

[1523] Input: Uploaded photo data

[1524] Output: Analyzed material information

[1525] Specific operation: The server uses Keras image recognition model to analyze the photo data and identify the ingredients contained in the dish. As a result of the analysis, it generates ingredient information.

[1526] Step 3:

[1527] The server detects whether or not an allergen is present based on the analysis results and notifies the user.

[1528] Input: Analyzed material information

[1529] Output: Notification of allergen presence / absence

[1530] Specific operation: The server detects whether the food contains an allergen based on the analyzed ingredient information. If so, it sends a notification to the user.

[1531] Step 4:

[1532] The server calculates the estimated calories based on the analysis results and displays them to the user.

[1533] Input: Analyzed material information

[1534] Output: Estimated calories displayed

[1535] Specific operation: The server calculates the calories of each ingredient based on the ingredient information, calculates the total calories, and displays the calculated calorie information to the user.

[1536] Step 5:

[1537] The server recognizes the user's emotions from their facial expressions and tone of voice.

[1538] Input: User facial expressions and tone of voice

[1539] Output: Recognized emotion information

[1540] Specific operation: The server uses the EmotionRecognizer library to analyze the user's facial expressions and tone of voice to recognize emotions, and generates recognized emotion information.

[1541] Step 6:

[1542] Servers suggest additional dishes and drinks based on perceived emotions.

[1543] Input: Recognized emotion information

[1544] Output: Additional food and drink suggestions

[1545] Specific operation: Based on the recognized emotion information, the server suggests dishes and drinks that suit the user's emotion and notifies the user of the suggestions.

[1546] Step 7:

[1547] The server stores the analysis results and survey results in a database and uses them to make meal suggestions for future visits.

[1548] Input: Analysis results, survey results

[1549] Output: Saved data, suggestions for future use

[1550] Specific operation: The server stores the analysis results and survey results in a database, and uses this data to make meal suggestions for the next time and beyond. The saved data is used to make suggestions that are appropriate for the user.

[1551] Example 2

[1552] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1553] Conventional meal recommendation systems can take into account participants' preferences and allergy information, but they have the problem of not being able to make suggestions that reflect participants' emotions or real-time situations. It is also difficult to determine whether the suggested dishes match the preferences of all participants, making it difficult to increase meal satisfaction. Furthermore, the system lacked the functionality to utilize past data to make suggestions for future meals, making it difficult to continuously improve satisfaction.

[1554] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1555] In this invention, the server includes a means for taking photos of meals, a means for analyzing the ingredients contained in the dishes from the photos, a means for sending information on the presence or absence of allergens, estimated calories, and advice on recommended drinks and foods to accompany the dishes based on the analysis results, a means for conducting a survey based on participants' preferences, a means for suggesting recommended dishes based on the survey results, a means for reading participants' emotions and suggesting meals that match those emotions, and a means for inputting prompts into a generative AI model to generate optimal dish suggestions. This enables real-time meal suggestions that take participants' preferences and emotions into account, thereby increasing meal satisfaction. Furthermore, by utilizing past data to make suggestions for future meals, continuous improvement in satisfaction is possible.

[1556] "Means for taking photos of food" refers to devices or functions for taking photos of food. Specifically, this applies to cameras and the camera functions of smartphones.

[1557] "Means for analyzing ingredients in a dish" refers to devices and software that identify and analyze ingredients in a dish from a photograph. Specifically, image recognition technology and ingredient databases are used.

[1558] "Means of sending advice on whether or not there are allergens, estimated calories, and recommended drinks and foods to ingest together" refers to devices and software that provide information on whether or not there are allergens and calories based on the analysis results, and give advice on appropriate foods and drinks.

[1559] "Means for surveying participants based on their preferences" refers to devices or software used to collect participants' dietary preferences or restrictions, such as survey forms or applications.

[1560] "Means for suggesting recommended dishes based on survey results" refers to devices or software that analyze the collected survey results and suggest the best dishes for participants.

[1561] "Means to read participants' emotions and suggest meals that match those emotions" refers to devices or software that analyze emotions from participants' facial expressions, voices, etc., and suggest appropriate meals based on those emotions. Specifically, this applies to emotion engines and facial expression recognition technology.

[1562] "Means for inputting prompt sentences into a generative AI model and generating optimal dish suggestions" refers to devices or software that input prompt sentences into a generative AI model and generate optimal dish suggestions as a result. Specifically, it uses natural language processing technology.

[1563] "Means for saving analysis results and survey results and making meal suggestions for the next time and beyond based on those results" refers to devices and software that save analysis results and survey results in a database and make meal suggestions for the next time and beyond based on that data.

[1564] This invention is a system that suggests the best dishes for a group meal based on the preferences and emotions of the participants. The system uses a questionnaire and an emotion engine to analyze the preferences and emotions of the participants and suggests dishes based on that.

[1565] Hardware and software used

[1566] Hardware: Servers, devices (smartphones, tablets, PCs)

[1567] Software: Survey interface, sentiment engine, database, generative AI model

[1568] Specific operation of the system

[1569] 1. Conducting a survey

[1570] The user uses a terminal to access the questionnaire interface provided by the system.

[1571] The user inputs their preferences and dietary restrictions into the questionnaire interface, such as "I like fish dishes," "I'm a vegetarian," "I like spicy food," "I don't have allergies," and "I like Japanese food."

[1572] The terminal transmits the inputted questionnaire data to the server in real time.

[1573] 2. Analysis of the survey data

[1574] The server stores the received questionnaire data in a database.

[1575] The server analyzes the survey data stored in the database and compiles the preferences of all participants. For example, if it turns out that everyone likes fish, it will use that information to move on to the next step.

[1576] The server then creates a prompt to be input to the generative AI model based on the analysis results. For example, it generates a prompt such as "fish dishes, vegetarian, spicy dishes, no allergies, Japanese food."

[1577] 3. Food suggestions

[1578] The server inputs a prompt sentence into the generative AI model.

[1579] The generative AI model generates optimal dish suggestions based on the prompt, for example, "a Japanese-style fish dish that is not spicy."

[1580] The server transmits the generated cooking suggestions to the terminal.

[1581] The user checks the suggested dishes through the terminal. For example, "a mild Japanese-style fish dish" is displayed on the terminal screen.

[1582] 4. Use of Emotion Engine (Alternative Embodiment)

[1583] The user sends their facial expressions to the system using the device's camera, for example, showing a happy expression while eating.

[1584] The terminal transmits the facial expression data to the server in real time.

[1585] The server uses an emotion engine to analyze the facial expression data and interpret the emotions of all participants. For example, it recognizes that everyone is looking happy.

[1586] The server generates a prompt based on the emotion to be input to the generative AI model, for example, "A dish that matches a fun atmosphere."

[1587] The generative AI model generates emotion-matching food suggestions based on the prompt, for example, "Party Platter."

[1588] The server transmits the generated cooking suggestions to the terminal.

[1589] The user checks the proposed dishes through the terminal. For example, "Party Platter" is displayed on the terminal screen.

[1590] Specific examples

[1591] For example, if there are five participants, the system works as follows:

[1592] 1. Conducting a survey

[1593] User A enters "I like fish dishes."

[1594] User B enters "vegetarian."

[1595] User C enters "I like spicy food."

[1596] User D enters "No allergies."

[1597] User E enters "I like Japanese food."

[1598] 2. Analysis of the survey data

[1599] The server aggregates this data and generates prompts to suggest dishes that will satisfy everyone.

[1600] 3. Food suggestions

[1601] The generative AI model is given the prompt sentence "Fish dish, vegetarian, spicy dish, no allergies, Japanese food."

[1602] The generative AI model suggests "non-spicy Japanese-style fish dishes."

[1603] The server sends this suggestion to the terminal, and the user confirms the suggested dish.

[1604] 4. Utilizing the Emotion Engine

[1605] If the user shows a happy expression, the emotion engine will recognize that happiness.

[1606] The generative AI model is given the prompt, "Food that matches a fun atmosphere."

[1607] The generative AI model suggests "a platter of food for a party."

[1608] The server sends this suggestion to the terminal, and the user confirms the suggested dish.

[1609] The above is a specific embodiment of this system.

[1610] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1611] Step 1:

[1612] Conducting a survey

[1613] The user uses a terminal to access the questionnaire interface provided by the system.

[1614] The user inputs their preferences and dietary restrictions into the questionnaire interface, such as "I like fish dishes," "I'm a vegetarian," "I like spicy food," "I don't have allergies," and "I like Japanese food."

[1615] Input: User preferences and dietary restrictions

[1616] The terminal transmits the inputted questionnaire data to the server in real time.

[1617] Output: Survey data sent to the server

[1618] Step 2:

[1619] Analysis of survey data

[1620] The server stores the received questionnaire data in a database.

[1621] The server analyzes the survey data stored in the database and compiles the preferences of all participants. For example, if it turns out that everyone likes fish, it will use that information to move on to the next step.

[1622] Input: Survey data stored in a database

[1623] The server then creates a prompt to be input to the generative AI model based on the analysis results. For example, it generates a prompt such as "fish dishes, vegetarian, spicy dishes, no allergies, Japanese food."

[1624] Output: Prompt sentence to be input to the generative AI model

[1625] Step 3:

[1626] Cooking suggestions

[1627] The server inputs a prompt sentence into the generative AI model.

[1628] The generative AI model generates optimal dish suggestions based on the prompt, for example, "a Japanese-style fish dish that is not spicy."

[1629] Input: The prompt given to the generative AI model

[1630] The server transmits the generated cooking suggestions to the terminal.

[1631] Output: Food suggestions sent to device

[1632] The user checks the suggested dishes through the terminal. For example, "a mild Japanese-style fish dish" is displayed on the terminal screen.

[1633] Step 4:

[1634] Use of Emotion Engine (another embodiment)

[1635] The user sends their facial expressions to the system using the device's camera, for example, showing a happy expression while eating.

[1636] Input: User's facial expression data

[1637] The terminal transmits the facial expression data to the server in real time.

[1638] The server uses an emotion engine to analyze the facial expression data and interpret the emotions of all participants. For example, it recognizes that everyone is looking happy.

[1639] Input: facial expression data

[1640] The server generates a prompt based on the emotion to be input to the generative AI model, for example, "A dish that matches a fun atmosphere."

[1641] Output: Prompt sentence to be input to the generative AI model

[1642] The generative AI model generates emotion-matching food suggestions based on the prompt, for example, "Party Platter."

[1643] Input: The prompt given to the generative AI model

[1644] The server transmits the generated cooking suggestions to the terminal.

[1645] Output: Food suggestions sent to device

[1646] The user checks the proposed dishes through the terminal. For example, "Party Platter" is displayed on the terminal screen.

[1647] (Application example 2)

[1648] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1649] Conventional meal recommendation systems make suggestions based on participants' preferences and dietary restrictions, but they have the problem of not being able to make suggestions that are tailored to the participants' emotions or the atmosphere of the occasion. Furthermore, when dining with multiple people, it is difficult to suggest dishes that will satisfy everyone. Furthermore, there was a need for a system that can easily make these suggestions using devices such as smartphones or smart glasses.

[1650] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1651] In this invention, the server includes means for acquiring images of meals, means for analyzing the ingredients contained in the dishes from the acquired images, means for providing the presence or absence of allergens, estimated calories, and advice on recommended drinks and foods to ingest based on the analysis results, means for conducting a questionnaire based on the preferences of participants in the case of a meal with multiple people, means for suggesting recommended dishes based on the questionnaire results, and means for reading the emotions of participants using an emotion engine and suggesting optimal dishes based on those emotions. This makes it possible to suggest optimal dishes based on the preferences and emotions of all participants.

[1652] The "means for acquiring an image of a meal" refers to a device or function for taking an image of a meal using a camera or the camera function of a smartphone and acquiring the image data.

[1653] "Means for analyzing the ingredients contained in a dish from an acquired image" refers to a device or function that uses image analysis technology to identify the ingredients contained in a dish from a photographed image of the food and extract that information.

[1654] "Means for providing advice on the presence or absence of allergens, estimated calories, and recommended drinks and foods to be consumed together based on the analysis results" refers to a device or function that calculates the presence or absence of allergens and estimated calories based on the results of ingredient analysis, and also provides advice on recommended drinks and foods to be consumed together.

[1655] "Means for conducting a questionnaire based on the preferences of participants when dining with multiple people" refers to a device or function for collecting the dietary preferences and restrictions of multiple participants in the form of a questionnaire.

[1656] The "means for proposing recommended dishes based on survey results" refers to a device or function for proposing dishes that will satisfy all participants based on the collected survey results.

[1657] "Means for reading participants' emotions using an emotion engine and suggesting the most suitable dish based on those emotions" refers to a device or function that uses emotion recognition technology to read emotions from participants' facial expressions and tone of voice, and suggests the most suitable dish based on those emotions.

[1658] "Means for saving analysis results and survey results and making meal suggestions for the next time and beyond based on those results" refers to a device or function that saves past analysis results and survey results in a database and makes meal suggestions for the next time and beyond based on that data.

[1659] "Applications installed on smartphones, smart glasses, head-mounted displays, or robots" are software applications that are installed on these devices and designed to be easily accessible to users.

[1660] This invention is a system that suggests optimal dishes for a group meal based on the preferences and emotions of the participants. This system is implemented as an application installed on a smartphone, smart glasses, a head-mounted display, or a robot.

[1661] System configuration

[1662] The system consists of the following main components:

[1663] 1. Method for acquiring food images: Take a picture of the food using the camera function of a smartphone or smart glasses.

[1664] 2. A method for analyzing the ingredients contained in the food from the captured image: Using image analysis technology, the ingredients contained in the food are identified from the captured image. Specifically, an image processing library such as OpenCV is used.

[1665] 3. A means of providing advice on the presence or absence of allergens, estimated calories, and recommended beverages and foods to consume together based on the analysis results: Based on the results of the ingredient analysis, the presence or absence of allergens and estimated calories are calculated, and advice is provided on the recommended beverages and foods to consume together.

[1666] 4. For group meals, a survey based on participants' preferences is conducted: A survey is conducted to collect information on each participant's dietary preferences and restrictions. The survey is conducted through a smartphone application interface.

[1667] 5. A method for suggesting recommended dishes based on survey results: Based on the collected survey results, dishes that will satisfy all participants are suggested.

[1668] 6. Using an emotion engine to read participants' emotions and suggest the best dish based on those emotions: Using emotion recognition technology, we read participants' emotions from their facial expressions and tone of voice, and suggest the best dish based on those emotions. Specifically, we use the EmotionRecognizer library.

[1669] 7. A means of saving analysis results and survey results and making meal suggestions for the next time and beyond based on those results: Past analysis results and survey results are saved in a database, and meal suggestions for the next time and beyond are made based on that data.

[1670] Processing flow

[1671] 1. The user takes a picture of the meal using the smartphone camera.

[1672] 2. The device analyzes the captured image and identifies the ingredients in the dish.

[1673] 3. Based on the analysis results, the device will calculate the presence or absence of allergens and estimated calories, and also provide advice on recommended beverages and foods to consume.

[1674] 4. If the user is dining with multiple people, collect the preferences and dietary restrictions of all participants through a survey.

[1675] 5. The device will suggest recommended dishes based on the survey results.

[1676] 6. The device uses an emotion engine to read the participant's emotions and suggests the best dish based on those emotions.

[1677] 7. The server stores the analysis results and survey results and uses them to make meal suggestions for future visits.

[1678] Specific examples

[1679] For example, at a family dinner, the app can suggest dishes that suit everyone's tastes and mood that day, or at a party with friends, it can suggest dishes that everyone will enjoy.

[1680] Prompt Sentence Examples

[1681] "If you're having a family dinner, suggest dishes that fit everyone's tastes and their mood that day. For example, if everyone likes fish and looks happy, what dishes would you suggest?"

[1682] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1683] Step 1:

[1684] The user takes a picture of the meal using the smartphone camera.

[1685] Input: Food image

[1686] Output: Captured image data

[1687] Specific operation: The user launches the camera app on their smartphone and takes a picture of their meal. The captured image is saved in the application.

[1688] Step 2:

[1689] The device analyzes the captured image and identifies the ingredients contained in the dish.

[1690] Input: Photographed image data

[1691] Output: Information on ingredients contained in the dish

[1692] Specific operation: The device uses image processing libraries such as OpenCV to analyze the ingredients of the food from the captured image. As a result of the analysis, the main ingredients contained in the food are identified.

[1693] Step 3:

[1694] Based on the analysis results, the device calculates the presence or absence of allergens and estimated calories, and also provides advice on recommended beverages and foods to consume.

[1695] Input: Information about ingredients contained in the dish

[1696] Output: Presence or absence of allergens, estimated calories, and recommended drinks and foods

[1697] How it works: The device checks for allergens based on ingredient information, calculates estimated calories, and generates recommendations for drinks and foods to ingest based on the ingredients.

[1698] Step 4:

[1699] If a user is dining with multiple people, the preferences and dietary restrictions of all participants are collected through a questionnaire.

[1700] Input: Survey responses regarding participants' preferences and dietary restrictions

[1701] Output: Survey result data

[1702] Specific operation: The user uses the survey function of the smartphone application to enter the preferences and dietary restrictions of all participants. The survey results are saved within the application.

[1703] Step 5:

[1704] The device will suggest recommended dishes based on the survey results.

[1705] Input: Survey result data

[1706] Output: Recommended dish list

[1707] Specific operation: The device analyzes the survey results and suggests dishes that will satisfy all participants. The suggested dishes are then displayed to the user.

[1708] Step 6:

[1709] The device uses an emotion engine to read the participant's emotions and suggests the best dish based on those emotions.

[1710] Input: Participant's facial expressions and tone of voice

[1711] Output: Emotion data, best recipe list

[1712] How it works: The device uses the EmotionRecognizer library to analyze participants' facial expressions and tone of voice via the camera and microphone. Based on the emotional data, it suggests the best dishes to serve.

[1713] Step 7:

[1714] The server stores the analysis results and survey results and uses them to make meal suggestions for future visits.

[1715] Input: Analysis result data, questionnaire result data

[1716] Output: Saved data, meal suggestions for the next time

[1717] Specific operation: The server stores the analysis results and survey results in a database and accumulates data to be used for future meal suggestions.

[1718] Example 3

[1719] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1720] Conventional meal recommendation systems do not take into account the user's eating history or emotional state when making suggestions, and therefore do not contribute sufficiently to improving the user's health management or satisfaction. Furthermore, when dining with multiple people, it is difficult to make suggestions that take into account the preferences and allergy information of all participants.

[1721] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1722] In this invention, the server includes: a means for taking photos of meals; a means for analyzing the ingredients contained in the meals from the photos; a means for sending, based on the analysis results, information on the presence or absence of allergens, estimated calories, and advice on recommended accompanying foods and drinks; a means for inputting the user's meal data and questionnaire results; a means for saving data; a means for analyzing the saved data; a means for generating next meal suggestions based on the analysis results; a means for displaying the generated meal suggestions to the user; and a means for analyzing the user's emotions using an emotion analysis engine and making meal suggestions based on the results. This enables meal suggestions to be made taking into account the user's meal history and emotional state, contributing to the user's health management and improved satisfaction. Furthermore, even for meals with multiple people, suggestions can be made taking into account the preferences and allergy information of all participants.

[1723] "Means for taking photos of food" refers to devices or functions that allow users to take photos of food. Specifically, this applies to smartphones and digital cameras.

[1724] "Means for analyzing ingredients contained in a dish from a photograph" refers to technologies and algorithms that analyze photographs of food and identify the ingredients contained therein, such as image analysis software and machine learning models.

[1725] "A means of sending advice on whether or not there are any allergens, estimated calories, and recommended drinks and foods to ingest" refers to a function that provides users with information on whether or not there are any allergens, calorie information, and suggestions for appropriate drinks and foods based on the analysis results.

[1726] "Means for inputting user's dietary data and survey results" refers to an interface or device that allows users to input their dietary details and survey results. Specifically, this applies to smartphone apps and web forms.

[1727] "Means for storing data" refers to a database or storage system for storing the meal data and survey results entered by users.

[1728] "Means for analyzing stored data" refers to technologies and algorithms for analyzing a user's dietary history and health status based on stored data, such as data analysis software and machine learning models.

[1729] "Means for generating next meal suggestions based on analysis results" refers to a function for making next meal suggestions to the user based on the analysis results. Specifically, this applies to a suggestion algorithm or recommendation engine.

[1730] The "means for displaying the generated meal suggestions to the user" refers to an interface or device for displaying the generated meal suggestions to the user, such as a smartphone app or a web page.

[1731] "Means of analyzing user emotions using an emotion analysis engine and making meal suggestions based on the results" refers to the technology and algorithms used to analyze user emotions and the functionality used to make meal suggestions based on the results. Emotion analysis software and machine learning models fall into this category.

[1732] MODE FOR CARRYING OUT THE INVENTION

[1733] The present invention is a system for making meal suggestions that take into account a user's eating history and emotional state. Specific embodiments of this system will be described below.

[1734] System configuration

[1735] The system consists of the following main components:

[1736] 1. Terminal: A device that users use to input their dietary data and survey results. Specifically, this applies to smartphones and PCs.

[1737] 2. Server: The central component responsible for data storage, analysis, and recommendation generation. It includes a database server and a sentiment analysis engine.

[1738] 3. Database: A storage system for saving user dietary data and survey results. Specifically, this is a database management system such as MySQL.

[1739] 4. Emotion analysis engine: Software for analyzing user emotions. Specifically, Microsoft Azure Emotion API is one example.

[1740] Program processing

[1741] The server receives the meal data and questionnaire results entered by the user from their device and stores them in a database. The stored data is referenced when making the next meal suggestion. The server retrieves the user's meal history from the database and analyzes it. Based on the analysis results, it generates a next meal suggestion and sends it to the user's device.

[1742] Furthermore, a sentiment analysis engine is used to analyze the user's emotions. For example, if the user shows a happy expression after eating a particular dish, this information is saved and used as a reference when suggesting a meal next time.

[1743] Specific examples

[1744] As a concrete example, let's say a user had "pizza" for their last meal. If the user's calorie intake was high at that time, the next time the system suggests a meal, it will suggest "salad" or "low-calorie soup." Also, if the user shows a happy expression after eating pizza, that information will be saved and the next time it suggests "pizza" or a "similar dish."

[1745] Prompt Sentence Examples

[1746] An example of a prompt to input to a generative AI model is as follows:

[1747] Generate a program that will suggest the next meal based on the user's eating history and the results of emotion analysis. If the user last ate a high-calorie meal, suggest a low-calorie meal, and if the user showed a happy expression after eating a particular dish, suggest that dish again.

[1748] In this way, the server realizes a system that makes meal suggestions for the next time and thereafter based on the user's meal history and emotion analysis results. The flow of the specification process in the third embodiment will be described with reference to FIG.

[1749] Program processing flow

[1750] Step 1: Entering user's dietary data and survey results

[1751] After finishing a meal, users use a device (smartphone or PC) to input information such as the meal contents, calories, and satisfaction level. They also enter data such as their impressions and physical condition in a questionnaire format.

[1752] Input: Meal details, calories, satisfaction, impressions, physical condition

[1753] Output: Sending input data

[1754] Specific behavior:

[1755] The user opens the dedicated application and inputs the meal details.

[1756] The user answers a questionnaire and inputs their level of satisfaction and physical condition.

[1757] The terminal transmits the input data to the server.

[1758] Step 2: Server saves data

[1759] The server stores the meal data and questionnaire results entered by the user in a database. The stored data is organized for each user and is referenced the next time meal suggestions are made.

[1760] Input: Meal data and survey results submitted by the user

[1761] Output: Save to database

[1762] Specific behavior:

[1763] The server receives the data sent by the user.

[1764] The server stores the received data in a database.

[1765] Step 3: Parse the data on the server

[1766] The server analyzes the stored data and evaluates the user's dietary balance and emotional state. For example, if the user eats a high-calorie meal, the server records that information and reflects it in the next meal recommendation.

[1767] Input: Dietary data stored in the database and questionnaire results

[1768] Output: Analysis results (calorie intake, satisfaction rating, etc.)

[1769] Specific behavior:

[1770] The server retrieves the user's meal history from the database.

[1771] The server analyzes the acquired data and evaluates calorie intake and satisfaction.

[1772] Step 4: Server generates next meal suggestions

[1773] The server generates meal suggestions based on the analysis results. For example, it will suggest low-calorie meals to a user who previously ate a high-calorie meal. It also uses an emotion engine to suggest dishes that made the user happy.

[1774] Input: Analysis results, emotion analysis results

[1775] Output: Next meal suggestion

[1776] Specific behavior:

[1777] The server generates meal suggestions based on the analysis results.

[1778] The server uses an emotion engine to analyze the user's emotion data and reflects it in the proposal.

[1779] Step 5: Displaying meal suggestions to the user

[1780] The server sends the generated meal suggestions to the user's terminal, where the user can check the next meal suggestions.

[1781] Input: Next meal suggestion

[1782] Output: Proposal displayed on user's device

[1783] Specific behavior:

[1784] The server transmits the generated meal suggestions to the user's terminal.

[1785] The user checks the suggested meals on the terminal.

[1786] In this way, the server realizes a system that makes meal suggestions for the next time and beyond based on the user's meal history and emotion analysis results.

[1787] (Application example 3)

[1788] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1789] Conventional meal recommendation systems often make suggestions based on the user's eating history and survey results, but they lack suggestions that take into account the user's emotions and past eating history. Furthermore, when dining with multiple people, it is difficult to make suggestions that reflect the preferences of all participants. Furthermore, there is a need for a system that can easily make these suggestions using devices such as smartphones or smart glasses.

[1790] The identification process performed by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a means for taking a photo of the meal, a means for analyzing the ingredients contained in the dish from the taken photo, a means for sending information on the presence or absence of allergens, estimated calories, and advice on recommended drinks and foods to accompany the meal based on the analysis results, a means for saving the analysis results and questionnaire results and making meal suggestions based on those results, and a means for reading the user's emotions using an emotion engine and making meal suggestions tailored to those emotions. This enables meal suggestions that take into account the user's emotions and past eating history, and even in cases of multiple people, suggestions that reflect the preferences of all participants. Furthermore, a system can be realized that allows these suggestions to be easily accessed using devices such as smartphones and smart glasses.

[1791] "Means for taking photos of meals" refers to camera functions or applications that allow users to take photos of their meals.

[1792] "Means for analyzing ingredients contained in a dish from a photograph" refers to image analysis technology that analyzes photographs of food and identifies the ingredients and components contained therein.

[1793] "A means of sending advice on whether or not there are any allergens, estimated calories, and recommended drinks and foods to ingest based on the analysis results" refers to a system that uses analyzed dietary information to determine whether or not there are any allergens, calculate calories, and even suggest appropriate drinks and foods.

[1794] "A means of saving analysis results and survey results, and making meal suggestions for the next time and beyond based on those results" refers to a system that saves a user's meal history and survey results in a database and makes meal suggestions for the next time based on those results.

[1795] "A means of reading a user's emotions using an emotion engine and suggesting meals that match those emotions" refers to technology that analyzes emotions from a user's facial expressions and behavior and suggests appropriate meals based on those emotions.

[1796] "A means of conducting a survey based on participants' preferences when dining with multiple people" refers to a system that collects information on each user's preferences and allergies in the form of a survey when multiple users are dining together.

[1797] "Means for suggesting recommended dishes based on survey results" refers to a system for suggesting dishes suitable for all participants based on the collected survey results.

[1798] "Application installed on a smartphone, smart glasses, head-mounted display, or robot" refers to software that is installed on a device used by a user and that runs the meal recommendation system.

[1799] The following system configuration will be described as an embodiment of the present invention.

[1800] System Configuration

[1801] The system can be used by users using smartphones, smart glasses, head-mounted displays, or robots. The system consists of the following main components:

[1802] 1. A way to take photos of your food

[1803] Users take photos of their meals using the camera function on their smartphone or smart glasses.

[1804] 2. A method for analyzing ingredients contained in a dish from a photograph

[1805] The server uses image analysis software to analyze the photos and identify the ingredients in the dish, using image recognition technology and machine learning models.

[1806] 3. A method to send advice on whether or not you have any allergies, estimated calories, and recommended drinks and foods to ingest based on the analysis results

[1807] Based on the analysis results, the server calculates whether or not there are any allergens and the estimated calories, and sends the user advice on appropriate drinks and foods.

[1808] 4. A means to save analysis results and survey results and make meal suggestions based on those results for future meals.

[1809] The server stores the user's meal history and survey results in a database and refers to this data when making the next meal suggestion.

[1810] 5. A method to read the user's emotions using an emotion engine and suggest meals that match those emotions.

[1811] The server uses an emotion engine to analyze the user's emotions from their facial expressions and behavior, and then suggests appropriate meals based on those emotions.

[1812] 6. For group meals, survey participants based on their preferences.

[1813] When dining with multiple people, the server collects preferences and allergy information from all participants in a questionnaire format.

[1814] 7. A way to suggest dishes based on survey results

[1815] Based on the collected survey results, the server will suggest dishes suitable for all participants.

[1816] Processing flow

[1817] 1. Users take photos of their meals

[1818] Users take photos of their food using their smartphone or smart glasses.

[1819] 2. Photo Analysis

[1820] The server uses image analysis software to analyze the photos and identify the ingredients contained in the dish.

[1821] 3. Submitting Advice

[1822] Based on the analysis results, the server calculates whether or not there are any allergens and the estimated calories, and sends the user advice on appropriate drinks and foods.

[1823] 4. Data storage

[1824] The server stores the analysis results and survey results in a database and refers to this data when making the next meal suggestion.

[1825] 5. Emotion analysis

[1826] The server uses an emotion engine to analyze the user's emotions from their facial expressions and behavior, and then suggests appropriate meals based on those emotions.

[1827] 6. Collecting surveys with multiple people

[1828] When dining with multiple people, the server collects preferences and allergy information from all participants in a questionnaire format.

[1829] 7. Food suggestions

[1830] Based on the collected survey results, the server will suggest dishes suitable for all participants.

[1831] Specific examples

[1832] User A: Last time, he ordered a high-calorie pizza, and responded in the survey that "It was delicious, but I'm concerned about the calories." The emotion engine detects a "happy" facial expression.

[1833] Suggestion for next time: Suggest a low-calorie salad or grilled chicken.

[1834] Prompt Sentence Examples

[1835] User ID: 1

[1836] Last meal: Pizza

[1837] Survey results: It was delicious, but I was concerned about the calories.

[1838] Emotion data: Joy

[1839] Next meal suggestion: Low-calorie salad and grilled chicken

[1840] In this way, a system can be created that suggests next meal plans based on a user's past eating history and emotional data.

[1841] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1842] Step 1:

[1843] A user takes a photo of a meal using a smartphone or smart glasses. The input is the photo of the meal taken by the user, and the output is the photo data.

[1844] Step 2:

[1845] The device sends the photograph taken to the server. The input is the photograph data taken by the user, and the output is the photograph data sent to the server.

[1846] Step 3:

[1847] The server uses image analysis software to analyze the photos sent and identify the ingredients contained in the dish. The input is the photo data, and the output is the analyzed ingredient data. Specifically, the system uses image recognition technology and machine learning models to identify ingredients.

[1848] Step 4:

[1849] The server calculates the presence or absence of allergens and estimated calories based on the analysis results, and generates appropriate drink and food recommendations. The input is the analyzed ingredient data, and the output is allergy information, calorie information, and advice information. Specifically, it retrieves allergy information from the database and applies a calorie calculation algorithm.

[1850] Step 5:

[1851] The server saves the analysis results and survey results in a database. The input is the analysis results and survey results, and the output is the saved data. Specifically, it executes an SQL query to insert data into the database.

[1852] Step 6:

[1853] The server uses an emotion engine to analyze emotions from the user's facial expressions and behavior. The input is the user's facial expression and behavior data, and the output is the analyzed emotion data. The specific operation is to apply an emotion recognition algorithm.

[1854] Step 7:

[1855] The server suggests appropriate meals based on the emotion data. The input is the emotion data and the stored meal history data, and the output is meal suggestion data. Specifically, the server combines the emotion data and the meal history data and applies a meal suggestion algorithm.

[1856] Step 8:

[1857] When the server is hosting a meal for multiple people, it conducts a survey based on the participants' preferences. The input is the participants' preferences and allergy information, and the output is the survey result data. Specifically, it generates a survey form and sends it to the participants.

[1858] Step 9:

[1859] The server recommends dishes based on the survey results. The input is the survey result data, and the output is the recommended dish data. Specifically, the server analyzes the survey results and applies an algorithm to select the optimal dish.

[1860] Step 10:

[1861] The server sends the generated meal suggestions and advice to the user's device. The input is meal suggestion data and advice data, and the output is suggestions and advice displayed on the user's device. Specifically, the data is sent to the user's device and displayed in the application.

[1862] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1863] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1864] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1865] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1866] [Third embodiment]

[1867] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1868] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1869] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1870] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1871] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1872] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1873] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1874] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1875] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1876] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1877] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1878] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1879] "Example 1"

[1880] When a user takes a photo of a meal, the system analyzes the ingredients contained in the dish from the photo. The analysis is performed using image recognition technology, specifically a deep learning-based neural network, to identify the type of dish and its ingredients from the photo. Based on the analysis results, the system sends the user advice on whether the dish contains any allergens, estimated calories, and recommended drinks and foods to accompany the dish.

[1881] "Example 2"

[1882] For multi-person dining, the system of the present invention conducts a survey based on the participants' preferences. The survey is conducted through an interface provided by the system, and participants input their preferences and dietary restrictions. Based on the survey results, the system recommends dishes. For example, if the survey reveals that all participants prefer fish dishes, the system will suggest a menu centered around fish dishes.

[1883] "Example 3"

[1884] The system of the present invention saves the analysis results and questionnaire results and makes meal suggestions based on those results for the next meal. The results are saved in a database within the user's profile and are referenced when making the next meal suggestion. For example, if a user ate a high-calorie meal in their last meal, the system can suggest a low-calorie meal for the next meal, taking into account the user's dietary balance.

[1885] The processing flow of each embodiment will be described below.

[1886] "Example 1"

[1887] Step 1: The user takes a photo of their meal.

[1888] Step 2: The system analyzes the ingredients in the dish from the photo using a deep learning-based neural network.

[1889] Step 3: Based on the analysis results, the system will send the user advice on whether they have any allergies, estimated calories, and recommended drinks and foods to ingest together.

[1890] "Example 2"

[1891] Step 1: For multi-person dining, the system surveys participants based on their preferences.

[1892] Step 2: Participants complete a questionnaire, including their preferences and dietary restrictions.

[1893] Step 3: Based on the survey results, the system will suggest recommended dishes.

[1894] "Example 3"

[1895] Step 1: The system saves the analysis results and survey results.

[1896] Step 2: The saved results will be referenced when suggesting meals from the next time onwards.

[1897] Step 3: The next time a meal recommendation is made, the system will use the saved results to make the recommendation.

[1898] Example 1

[1899] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1900] Conventional meal management systems require users to manually enter meal details, which is time-consuming and often lacks accuracy. They also struggle to provide comprehensive meal management, such as checking for allergens, calculating calories, and providing advice on accompanying foods and drinks. Furthermore, when dining with multiple people, they are unable to provide suggestions that take into account the preferences and allergy information of each participant. To address these issues, there is a need for a system that allows users to easily manage meal details and provides accurate allergy information, calorie calculations, and appropriate advice.

[1901] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1902] In this invention, the server includes a means for acquiring an image of a meal, a means for analyzing the ingredients contained in the dish from the acquired image, a means for determining the presence or absence of allergens based on the analysis results, a means for calculating estimated calories based on the analysis results, and a means for providing advice on recommended foods and beverages to ingest based on the analysis results. This allows users to easily manage their meal contents and receive accurate allergy information, calorie calculations, and appropriate advice simply by taking a photo of their meal. Furthermore, in the case of a meal with multiple people, the server further includes a means for conducting a survey based on the preferences of the participants and a means for suggesting recommended dishes based on the survey results, thereby enabling suggestions that take into account the preferences and allergy information of all participants. Furthermore, the server includes a means for saving the analysis results and survey results and making meal suggestions based on those results for future meals, facilitating ongoing dietary management.

[1903] "Means for acquiring images of meals" refers to devices or software that allow a user to take a photo of a meal and send the image data to the system.

[1904] "Means for analyzing the ingredients contained in a dish from an acquired image" refers to devices or software that use image recognition technology to identify the type of dish and its ingredients from a photograph of a meal.

[1905] "Means for determining whether or not an allergen is present" refers to devices or software that compares the analyzed material information with the allergy information registered by the user to confirm whether or not an allergen is present.

[1906] "Means for calculating estimated calories" refers to devices or software for calculating the total calories of a dish based on analyzed ingredient information.

[1907] "Means for providing advice on foods and beverages to be consumed together" refers to devices and software that suggest foods and beverages that go well with a dish based on the analysis results.

[1908] "Means for taking surveys" refers to devices or software used to collect information about preferences and allergies of participants at group meals.

[1909] "Means for suggesting recommended dishes" refers to devices or software that make recommendations based on the survey results, taking into account the preferences and allergy information of all participants.

[1910] "Means for saving analysis results and survey results" refers to devices and software for saving analyzed material information and survey results in a database or the like.

[1911] "Means for making meal suggestions for the next time and onwards" refers to devices or software for making meal suggestions for the next time and onwards based on the saved analysis results and survey results.

[1912] This invention is a system that allows a user to take a photo of a meal and analyzes the ingredients contained in the dish from the photo. The system includes means for acquiring an image of the meal, means for analyzing the ingredients contained in the dish from the acquired image, means for determining the presence or absence of allergens based on the analysis results, means for calculating estimated calories based on the analysis results, and means for providing advice on recommended foods and drinks to ingest along with the meal based on the analysis results.

[1913] Hardware and software used

[1914] Users take photos of their meals using a smartphone or camera-equipped device. The device then uploads the photos to the server. The server receives the uploaded photos and uses deep learning libraries such as TensorFlow and PyTorch to perform image analysis. The analysis results are stored in a database and notified to the user.

[1915] Specific examples

[1916] A user orders "curry rice" at a restaurant and takes a photo of it with their smartphone. The device uploads the photo to the server. The server receives the photo and performs image analysis using TensorFlow. As a result of the analysis, ingredients such as "rice, curry sauce, meat, and vegetables" are identified. The server compares this with the allergy information registered by the user to determine whether or not any allergens are present. For example, if the user has a "nut allergy," the server checks whether the ingredients contain nuts. Next, the server calculates the estimated calorie content of the dish based on the ingredients and displays "700 kcal." The server then generates advice such as "Lassi goes well with curry rice" and notifies the user.

[1917] Prompt Sentence Examples

[1918] "Analyze a user-taken photo of spaghetti bolognese and identify the ingredients. If the user has a dairy allergy, display a warning and provide food and drink recommendations along with estimated calories."

[1919] In this way, users can easily manage their meal contents, receive accurate allergy information, calorie calculations, and appropriate advice simply by taking photos of their meals. Furthermore, for meals with multiple people, including a means for conducting a survey based on the participants' preferences and a means for proposing recommended dishes based on the survey results makes it possible to make suggestions that take into account the preferences and allergy information of all participants. Furthermore, including a means for saving the analysis results and survey results and making future meal suggestions based on those results makes ongoing dietary management easier.

[1920] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1921] Step 1:

[1922] The user takes a photo of the meal.

[1923] A user takes a photo of a meal using a smartphone or other device with a camera. For example, a user orders "curry rice" at a restaurant and takes a photo of it. The input is the photo of the meal, and the output is the captured image data.

[1924] Step 2:

[1925] The device uploads the photo to the server.

[1926] The device sends the captured photos to a server via an Internet connection. The input is the captured image data, and the output is the image data uploaded to the server.

[1927] Step 3:

[1928] The server receives the photo and performs image analysis.

[1929] The server receives the photos sent from the device. It performs image analysis on the received photos using deep learning libraries such as TensorFlow and PyTorch. The input is the uploaded image data, and the output is the analyzed type of food and ingredient information. Specifically, the neural network model analyzes the photo and identifies "curry rice" and its ingredients.

[1930] Step 4:

[1931] The server identifies the material based on the analysis results.

[1932] The server identifies the type of food and its ingredients in the photo based on the results of image analysis. The input is the analyzed information on the type of food, and the output is the identified ingredient information. For example, if "curry rice" is identified, ingredients such as "rice, curry sauce, meat, and vegetables" will be listed.

[1933] Step 5:

[1934] The server checks for the presence of allergens.

[1935] The server compares the identified ingredients with the allergy information registered by the user. The input is the identified ingredient information and the user's allergy information, and the output is the determination result of whether or not the allergen is present. For example, if the user has a "nut allergy," it checks whether the ingredients contain nuts.

[1936] Step 6:

[1937] The server calculates the estimated calories.

[1938] The server calculates the estimated calories of a dish based on the identified ingredients. The input is the identified ingredient information, and the output is the calculated estimated calories. For example, in the case of "curry rice," the calories of each ingredient are added together to calculate "700 kcal."

[1939] Step 7:

[1940] The server then generates advice on recommended drinks and foods to consume.

[1941] Based on the analysis results, the server generates advice on recommended drinks and foods to ingest. The input is the analysis results and the identified ingredient information, and the output is the generated advice. For example, it generates advice such as "Lassi goes well with curry rice."

[1942] ...

Claims

1. a means for taking a photo of the user's meal; A means for analyzing ingredients contained in a dish from a photograph taken; a means for generating and presenting advice using a prompt sentence instructing the device to generate advice on the presence or absence of allergens, estimated calories, and recommended drinks or foods to be consumed together based on the results of the analysis, and a generative AI model; and A system including:

2. A means for storing the user's meal history or questionnaire results in a database; The system of claim 1 , further comprising: a means for analyzing the stored meal history or questionnaire results, evaluating the user's meal balance, and presenting the next meal suggestion using a prompt sentence instructing the user to make a next meal suggestion and the generative AI model.

3. means for recognizing an emotion of the user; The presenting means includes: The system of claim 1 , wherein the advice is generated and presented using a prompt sentence that instructs the advice to be generated based on the results of the analysis and the user's emotions, and the generative AI model.

Images